Jul 28 23:19:55.984204 ip-10-0-135-158 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 23:19:55.984213 ip-10-0-135-158 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 23:19:55.984221 ip-10-0-135-158 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 23:19:55.984436 ip-10-0-135-158 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 23:20:06.071006 ip-10-0-135-158 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 23:20:06.071024 ip-10-0-135-158 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3af648c12330467e96226845f87c920d --
Jul 28 23:21:44.400930 ip-10-0-135-158 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 23:21:44.741812 ip-10-0-135-158 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 23:21:44.741812 ip-10-0-135-158 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 23:21:44.741812 ip-10-0-135-158 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 23:21:44.741812 ip-10-0-135-158 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 23:21:44.741812 ip-10-0-135-158 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 23:21:44.742843 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.742749    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 23:21:44.746901 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746884    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 23:21:44.746901 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746902    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746906    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746909    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746913    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746916    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746919    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746922    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746925    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746928    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746931    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746935    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746938    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746941    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746947    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746950    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746953    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746956    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746958    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746961    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 23:21:44.746966 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746964    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746968    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746972    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746976    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746979    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746982    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746985    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746988    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746990    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746993    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.746997    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747000    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747003    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747007    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747010    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747013    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747016    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747019    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747021    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 23:21:44.747416 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747024    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747026    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747029    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747031    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747034    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747036    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747039    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747042    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747045    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747047    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747049    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747052    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747054    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747057    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747060    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747064    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747067    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747069    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747072    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747074    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 23:21:44.748022 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747077    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747080    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747082    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747085    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747087    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747092    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747095    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747098    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747101    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747104    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747107    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747110    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747112    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747115    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747118    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747121    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747124    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747127    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747129    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 23:21:44.748501 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747132    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747135    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747137    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747140    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747142    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747145    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747147    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747150    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747153    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747157    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747159    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747162    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747165    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747167    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747170    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747174    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747178    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747180    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747183    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747186    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 23:21:44.748963 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747188    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747929    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747936    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747939    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747943    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747946    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747950    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747952    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747955    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747958    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747961    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747964    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747966    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747969    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747971    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747974    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747976    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747979    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747983    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 23:21:44.749456 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747987    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747992    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747995    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.747998    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748000    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748003    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748006    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748008    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748011    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748013    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748016    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748018    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748021    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748024    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748027    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748030    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748032    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748035    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748038    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748041    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 23:21:44.749939 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748044    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748047    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748049    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748052    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748054    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748057    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748059    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748062    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748064    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748067    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748070    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748072    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748076    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748080    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748083    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748085    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748088    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748090    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748093    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748096    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 23:21:44.750423 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748098    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748101    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748103    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748106    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748108    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748111    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748113    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748116    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748119    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748123    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748126    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748128    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748131    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748134    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748136    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748139    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748142    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748144    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748146    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748149    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 23:21:44.750956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748152    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748154    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748157    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748159    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748162    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748170    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748174    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748176    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748179    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748181    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748184    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748187    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748190    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748192    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748195    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748197    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748200    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748202    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748205    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 23:21:44.751442 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748207    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748210    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748593    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748599    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748604    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748607    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748610    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748613    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748616    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748619    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748637    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748640    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748643    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748646    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748649    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748651    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748654    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748657    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748661    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748664    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 23:21:44.751919 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748667    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748670    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748672    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748675    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748678    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748681    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748684    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748686    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748689    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748692    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748695    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748698    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748700    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748703    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748705    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748708    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748711    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748714    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748716    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748720    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 23:21:44.752403 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748722    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748725    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748728    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748731    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748733    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748736    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748739    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748741    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748744    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748746    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748750    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748753    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748756    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748758    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748761    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748763    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748766    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748769    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748771    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748774    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 23:21:44.752890 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748776    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748779    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748781    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748785    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748788    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748791    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748793    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748796    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748798    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748802    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748806    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748809    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748812    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748817    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748820    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748822    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748825    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748827    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748830    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748832    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 23:21:44.753384 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748835    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748837    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748841    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748843    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748846    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748849    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748851    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748854    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748856    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748859    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748862    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748864    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748867    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748869    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748872    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748875    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748878    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748880    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748883    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 23:21:44.753888 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748885    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.748888    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.748969    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.748976    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749701    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749706    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749713    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749716    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749720    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749725    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749728    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749731    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749735    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749738    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749741    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749745    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749748    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749751    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749753    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749756    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749761    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749764    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749767    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 23:21:44.754326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749770    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749774    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749778    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749781    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749784    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749787    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749790    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749793    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749796    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749799    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749803    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749807    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749810    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749813    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749816    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749819    2581 flags.go:64] FLAG: --enable-server="true"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749823    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749828    2581 flags.go:64] FLAG: --event-burst="100"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749831    2581 flags.go:64] FLAG: --event-qps="50"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749834    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749837    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749840    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749844    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749846    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749850    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 23:21:44.754880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749853    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749856    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749859    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749862    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749865    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749869    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749872    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749875    2581 flags.go:64] FLAG: --feature-gates=""
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749879    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749882    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749885    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749888    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749892    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749894    2581 flags.go:64] FLAG: --help="false"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749897    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-135-158.ec2.internal"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749900    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749903    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749906    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749909    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749913    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749916    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749919    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749922    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 23:21:44.755474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749925    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749931    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749934    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749937    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749940    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749943    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749946    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749949    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749953    2581 flags.go:64] FLAG: --lock-file=""
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749956    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749959    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749962    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749968    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749971    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749974    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749977    2581 flags.go:64] FLAG: --logging-format="text"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749980    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749983    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749986    2581 flags.go:64] FLAG: --manifest-url=""
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749989    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749993    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.749997    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750001    2581 flags.go:64] FLAG: --max-pods="110"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750004    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750007    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 23:21:44.756089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750010    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750013    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750016    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750019    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750022    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750029    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750033    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750036    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750039    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750042    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750047    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750050    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750053    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750056    2581 flags.go:64] FLAG: --port="10250"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750060    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750062    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d5085d9ba050f9ee"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750066    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750069    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750071    2581 flags.go:64] FLAG: --register-node="true"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750074    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750078    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750081    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750084    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750087    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750090    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 23:21:44.756724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750093    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750096    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750099    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750102    2581 flags.go:64] FLAG: --runonce="false"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750105    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750108    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750111    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750114    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750117    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750120    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750123    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750126    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750129    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750134    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750137    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750140    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750143    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750146    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750149    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750154    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750158    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750160    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750165    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750169    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750172    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 23:21:44.757338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750175    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750179    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750181    2581 flags.go:64] FLAG: --v="2"
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750186    2581 flags.go:64] FLAG: --version="false"
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750190    2581 flags.go:64] FLAG: --vmodule=""
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750194    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.750197    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750314    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750318    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750322    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750325    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750328    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750331    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750334    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750336    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750339    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750342    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750345    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750348    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750351    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750355    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750358    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 23:21:44.757956 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750360    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750363    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750366    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750369    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750371    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750374    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750376    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750379    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750383    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750385    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750388    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750391    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750393    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750395    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750398    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750401    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750403    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750406    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750409    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750411    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 23:21:44.758490 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750414    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750417    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750420    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750422    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750425    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750427    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750430    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750433    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750437    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750441    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750445    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750448    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750450    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750453    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750455    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750458    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750461    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750464    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750466    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750469    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 23:21:44.758977 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750472    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750475    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750478    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750480    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750483    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750486    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750489    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750491    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750495    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750499    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750502    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750505    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750508    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750510    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750513    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750516    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750519    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750521    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750524    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 23:21:44.759463 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750526    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750529    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750532    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750536    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750539    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750541    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750544    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750546    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750549    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750552    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750555    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750558    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750560    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750564    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750567    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750571    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750575    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750578    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750581    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 23:21:44.759979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750584    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750586    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750589    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750592    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750595    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.750597    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.751365    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.758894    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.758910    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.759823    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.759833    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759891    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759896    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759899    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 23:21:44.760440 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759902    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759906    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759912    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759919    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759924    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759927    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759930    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759933    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759936    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759940    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759943    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759946    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759949    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759952    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759955    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759958    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759961    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759963    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759966    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759969    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 23:21:44.760803 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759971    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759974    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759976    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759979    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759983    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759987    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759991    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759996    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.759999    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760002    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760005    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760007    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760010    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760012    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760015    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760018    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760020    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760023    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760025    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760028    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 23:21:44.761288 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760031    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760033    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760036    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760039    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760042    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760045    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760047    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760050    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760052    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760055    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760057    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760060    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760063    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760067    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760072    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760076    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760080    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760083    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760085    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 23:21:44.761781 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760088    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760090    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760093    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760095    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760098    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760100    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760103    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760105    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760108    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760111    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760113    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760115    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760118    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760120    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760123    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760125    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760128    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760131    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760134    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 23:21:44.762240 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760136    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760139    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760142    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760148    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760153    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760157    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760160    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760162    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760165    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760167    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760170    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760172    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760174    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760177    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760180    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760182    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760187    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 23:21:44.762698 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760190    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.760195    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760342    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760348    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760351    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760354    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760356    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760359    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760361    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760364    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760367    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760369    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760372    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760375    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 23:21:44.763153 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760378    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760382    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760387    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760391    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760396    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760399    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760403    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760406    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760408    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760411    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760414    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760416    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760420    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760422    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760425    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760428    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760430    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760433    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760435    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 23:21:44.763496 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760438    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760441    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760445    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760448    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760451    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760453    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760456    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760459    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760463    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760468    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760472    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760476    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760479    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760482    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760485    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760487    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760490    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760492    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760495    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760498    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 23:21:44.764114 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760500    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760503    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760505    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760507    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760510    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760515    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760518    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760520    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760523    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760525    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760528    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760530    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760533    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760535    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760539    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760543    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760547    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760551    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760555    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 23:21:44.764587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760558    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760560    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760563    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760565    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760568    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760571    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760575    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760579    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760583    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760585    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760588    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760591    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760593    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760596    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760599    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760602    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760604    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760607    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760611    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760614    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 23:21:44.765048 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760617    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760621    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760638    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760643    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760647    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760650    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760653    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760655    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:44.760658    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.760663    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.760773    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.763185    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.764845    2581 server.go:1019] "Starting client certificate rotation"
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.764952    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 23:21:44.765539 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.764985    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 23:21:44.769373 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.769348    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 23:21:44.770557 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.770537    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 23:21:44.780822 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.780800    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 28 23:21:44.784890 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.784872    2581 log.go:25] "Validated CRI v1 image API"
Jul 28 23:21:44.785655 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.785642    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 23:21:44.788204 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.788182    2581 fs.go:135] Filesystem UUIDs: map[68375a52-30d9-41a0-aaf5-bd2d99cd7cc6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e66ba7c6-544d-4b0d-b577-fa8880ee2144:/dev/nvme0n1p4]
Jul 28 23:21:44.788265 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.788203    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 23:21:44.792974 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.792855    2581 manager.go:216] Machine: {Timestamp:2026-07-28 23:21:44.792343015 +0000 UTC m=+0.294889536 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099892 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284dfd8788980d486f2a1f70d69bce SystemUUID:ec284dfd-8788-980d-486f-2a1f70d69bce BootID:3af648c1-2330-467e-9622-6845f87c920d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0a:86:d9:58:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0a:86:d9:58:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:a5:e8:d6:b7:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 23:21:44.792974 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.792964    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 23:21:44.793140 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.793045    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 23:21:44.795460 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.795435    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 23:21:44.796948 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.796916    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 23:21:44.797094 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.796954    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-158.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 23:21:44.797813 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.797801    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 23:21:44.797860 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.797816    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 23:21:44.797860 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.797825    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 23:21:44.798809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.798798    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 23:21:44.800136 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.800124    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 23:21:44.803651 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.803618    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 23:21:44.803779 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.803769    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 23:21:44.803813 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.803804    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 23:21:44.803851 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.803818    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 23:21:44.803851 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.803842    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 23:21:44.803902 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.803856    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 28 23:21:44.803902 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.803869    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 23:21:44.804344 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.804332    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 23:21:44.804384 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.804357    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 23:21:44.807135 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.807119    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 23:21:44.811608 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.811585    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 23:21:44.811719 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.811642    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 23:21:44.813809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813792    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 23:21:44.813809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813813    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813820    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813826    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813837    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813846    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813859    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813866    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813876    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813885    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813895    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 23:21:44.813939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.813903    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 23:21:44.814205 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.813956    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 23:21:44.814205 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.813885    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-158.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 23:21:44.814831 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.814820    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 23:21:44.814911 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.814833    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 23:21:44.816656 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.816621    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-158.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 23:21:44.817970 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.817947    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wsnqt"
Jul 28 23:21:44.818138 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.818127    2581 server.go:1268] "Started kubelet"
Jul 28 23:21:44.818340 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.818212    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 23:21:44.818340 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.818229    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 23:21:44.818340 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.818307    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 23:21:44.819056 ip-10-0-135-158 systemd[1]: Started Kubernetes Kubelet.
Jul 28 23:21:44.820320 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.820304    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 23:21:44.820498 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.820480    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 23:21:44.823323 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.823298    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 23:21:44.824852 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.824835    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wsnqt"
Jul 28 23:21:44.831852 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.831835    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 23:21:44.831852 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.831844    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 23:21:44.831984 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.831944    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:44.832549 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832532    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 23:21:44.832549 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832532    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 23:21:44.832725 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832560    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 23:21:44.832725 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832715    2581 factory.go:55] Registering systemd factory
Jul 28 23:21:44.832836 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832732    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 28 23:21:44.832836 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832742    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 23:21:44.832836 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832749    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 23:21:44.832974 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832944    2581 factory.go:153] Registering CRI-O factory
Jul 28 23:21:44.832974 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.832959    2581 factory.go:223] Registration of the crio container factory successfully
Jul 28 23:21:44.833055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.833008    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 23:21:44.833055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.833029    2581 factory.go:103] Registering Raw factory
Jul 28 23:21:44.833055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.833044    2581 manager.go:1196] Started watching for new ooms in manager
Jul 28 23:21:44.833580 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.833564    2581 manager.go:349] Starting recovery of all containers
Jul 28 23:21:44.833768 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.833754    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 23:21:44.836044 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.836011    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-158.ec2.internal\" not found" node="ip-10-0-135-158.ec2.internal"
Jul 28 23:21:44.845898 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.843980    2581 manager.go:354] Recovery completed
Jul 28 23:21:44.850160 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.850143    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 23:21:44.852218 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.852195    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 23:21:44.852388 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.852236    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 23:21:44.852388 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.852251    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 23:21:44.852885 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.852868    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 23:21:44.852971 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.852885    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 23:21:44.852971 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.852899    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 23:21:44.852971 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.852906    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 23:21:44.853700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.853685    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 23:21:44.853769 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.853714    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 23:21:44.853769 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.853728    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 23:21:44.855701 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.855689    2581 policy_none.go:49] "None policy: Start"
Jul 28 23:21:44.855757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.855706    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 23:21:44.855757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.855717    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 23:21:44.857706 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.857695    2581 policy_none.go:47] "Start"
Jul 28 23:21:44.882268 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.882251    2581 manager.go:337] "Starting Device Plugin manager"
Jul 28 23:21:44.882372 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.882296    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 23:21:44.882372 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.882309    2581 server.go:87] "Starting device plugin registration server"
Jul 28 23:21:44.882611 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.882597    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 23:21:44.882683 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.882615    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 23:21:44.882779 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.882756    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 23:21:44.882891 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.882850    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 23:21:44.882891 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.882863    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 23:21:44.883327 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.883305    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 23:21:44.883421 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.883361    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:44.973736 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.973686    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 23:21:44.976051 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.975064    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 23:21:44.976051 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.975104    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 23:21:44.976051 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.975130    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 23:21:44.976051 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.975168    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 23:21:44.976844 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.976822    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 23:21:44.982913 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.982896    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 23:21:44.983734 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.983715    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 23:21:44.983855 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.983751    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 23:21:44.983855 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.983766    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 23:21:44.983855 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.983796    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-158.ec2.internal"
Jul 28 23:21:44.993788 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:44.993744    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-158.ec2.internal"
Jul 28 23:21:44.993788 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:44.993768    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-158.ec2.internal\": node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.006574 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.006553    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.075923 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.075880    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"]
Jul 28 23:21:45.076067 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.075975    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 23:21:45.076933 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.076918    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 23:21:45.077000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.076949    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 23:21:45.077000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.076962    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 23:21:45.079472 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.079457    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 23:21:45.079603 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.079571    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.079603 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.079604    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 23:21:45.080206 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.080188    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 23:21:45.080282 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.080219    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 23:21:45.080282 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.080229    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 23:21:45.080282 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.080189    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 23:21:45.080373 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.080291    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 23:21:45.080373 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.080301    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 23:21:45.082566 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.082546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.082676 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.082578    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 23:21:45.083278 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.083264    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 23:21:45.083334 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.083290    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 23:21:45.083334 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.083312    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeHasSufficientPID"
Jul 28 23:21:45.101468 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.101277    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-158.ec2.internal\" not found" node="ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.104832 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.104812    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-158.ec2.internal\" not found" node="ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.106850 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.106835    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.134036 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.134010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e097d8f7f5bef6b1daa1f2fc30e6048e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"e097d8f7f5bef6b1daa1f2fc30e6048e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.134181 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.134048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e097d8f7f5bef6b1daa1f2fc30e6048e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"e097d8f7f5bef6b1daa1f2fc30e6048e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.134181 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.134068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0594a384a49868ef7a8441f356bc9f2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-158.ec2.internal\" (UID: \"a0594a384a49868ef7a8441f356bc9f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.207192 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.207159    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.234546 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.234518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e097d8f7f5bef6b1daa1f2fc30e6048e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"e097d8f7f5bef6b1daa1f2fc30e6048e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.234604 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.234549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e097d8f7f5bef6b1daa1f2fc30e6048e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"e097d8f7f5bef6b1daa1f2fc30e6048e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.234604 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.234568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0594a384a49868ef7a8441f356bc9f2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-158.ec2.internal\" (UID: \"a0594a384a49868ef7a8441f356bc9f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.234683 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.234640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e097d8f7f5bef6b1daa1f2fc30e6048e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"e097d8f7f5bef6b1daa1f2fc30e6048e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.234719 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.234708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0594a384a49868ef7a8441f356bc9f2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-158.ec2.internal\" (UID: \"a0594a384a49868ef7a8441f356bc9f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.234751 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.234733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e097d8f7f5bef6b1daa1f2fc30e6048e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal\" (UID: \"e097d8f7f5bef6b1daa1f2fc30e6048e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.308056 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.307984    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.404571 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.404534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.407829 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.407809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:45.408231 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.408216    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.508982 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.508939    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.609512 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.609445    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.709940 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.709905    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.764379 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.764347    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 23:21:45.764847 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.764513    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 23:21:45.764847 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.764527    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 23:21:45.810800 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.810766    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.831000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.830971    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 23:16:45 +0000 UTC" deadline="2028-05-12 20:13:23.622273322 +0000 UTC"
Jul 28 23:21:45.831000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:45.830996    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15692h51m37.791280497s"
Jul 28 23:21:45.911368 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:45.911294    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:45.995905 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:45.995866    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0594a384a49868ef7a8441f356bc9f2.slice/crio-55a1a19c04a0aaaae1e590e677e5f319e0bd5433736e426b4f5fa6a541aa079b WatchSource:0}: Error finding container 55a1a19c04a0aaaae1e590e677e5f319e0bd5433736e426b4f5fa6a541aa079b: Status 404 returned error can't find the container with id 55a1a19c04a0aaaae1e590e677e5f319e0bd5433736e426b4f5fa6a541aa079b
Jul 28 23:21:45.996390 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:45.996369    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode097d8f7f5bef6b1daa1f2fc30e6048e.slice/crio-42c24685d4d1a5d350807993feb6b157e35c1c77f3a268cf22fa9afc4008c2e7 WatchSource:0}: Error finding container 42c24685d4d1a5d350807993feb6b157e35c1c77f3a268cf22fa9afc4008c2e7: Status 404 returned error can't find the container with id 42c24685d4d1a5d350807993feb6b157e35c1c77f3a268cf22fa9afc4008c2e7
Jul 28 23:21:46.001244 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.001226    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 23:21:46.012196 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.012171    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:46.112739 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.112676    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:46.175984 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.175918    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 23:21:46.214434 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.214381    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-158.ec2.internal\" not found"
Jul 28 23:21:46.265536 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.265507    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 23:21:46.333058 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.333024    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:46.343847 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.343825    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 23:21:46.344741 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.344727    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal"
Jul 28 23:21:46.352844 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.352819    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 23:21:46.779578 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.779544    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 23:21:46.804933 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.804897    2581 apiserver.go:52] "Watching apiserver"
Jul 28 23:21:46.812327 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.812285    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 23:21:46.813256 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.813228    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-m968q","kube-system/konnectivity-agent-86l88","kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk","openshift-network-diagnostics/network-check-target-cbw8q","openshift-network-operator/iptables-alerter-9xtbg","openshift-ovn-kubernetes/ovnkube-node-ms6kt","kube-system/global-pull-secret-syncer-fkq46","openshift-cluster-node-tuning-operator/tuned-wd8jd","openshift-dns/node-resolver-w5rtn","openshift-image-registry/node-ca-jvngb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal","openshift-multus/multus-additional-cni-plugins-8xxts","openshift-multus/multus-h9td4"]
Jul 28 23:21:46.816793 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.816767    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.819317 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.819295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.820627 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.820606    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 23:21:46.821608 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.821589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.822360 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.822338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 23:21:46.823344 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.823323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 23:21:46.823611 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.823592    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 23:21:46.823865 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.823846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gq66v\""
Jul 28 23:21:46.824072 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.824050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 23:21:46.824263 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.824248    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 23:21:46.824446 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.824429    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 23:21:46.824623 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.824606    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 23:21:46.824838 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.824821    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 23:21:46.825063 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.825046    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nx6z4\""
Jul 28 23:21:46.825258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.825237    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 23:21:46.825578 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.825540    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 23:21:46.825952 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.825929    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 23:21:46.826096 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.826060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qlptt\""
Jul 28 23:21:46.826168 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.826136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.826497 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.826480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:46.826583 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.826554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:21:46.828806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.828786    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 23:21:46.829063 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.828882    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 23:21:46.829063 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.829027    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xgjcc\""
Jul 28 23:21:46.830249 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.829173    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 23:21:46.830803 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.830782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.832431 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.832407    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 23:21:46.833227 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.833204    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 23:21:46.833321 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.833261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 23:21:46.833504 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.833480    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 23:21:46.833586 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.833564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.833723 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.833688    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnkm8\""
Jul 28 23:21:46.836155 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.836137    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 23:21:46.836398 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.836378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvz2b\""
Jul 28 23:21:46.836481 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.836420    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 23:21:46.836687 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.836668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:46.836788 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.836743    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:21:46.839593 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.839569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:46.841976 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.841943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-systemd\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.842096 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.841981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-run\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.842096 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-system-cni-dir\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.842096 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.842096 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-netns\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-var-lib-kubelet\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842119    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-run-netns\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-var-lib-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysconfig\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-log-socket\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842256    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 23:21:46.842300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovn-node-metrics-cert\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-etc-kubernetes\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-socket-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-sys-fs\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-iptables-alerter-script\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842404    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bxrhq\""
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-host\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42cf06e5-5e66-4b39-9671-d88da0a957db-cni-binary-copy\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-conf-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-daemon-config\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.842563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-host-slash\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-etc-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-k8s-cni-cncf-io\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-slash\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-cni-bin\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovnkube-config\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-cni-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.842962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.842871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-cni-bin\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-systemd-units\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-ovn\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-cni-multus\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-hostroot\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-kubelet\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-cni-netd\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-socket-dir-parent\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-registration-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.843451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-modprobe-d\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-lib-modules\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-cnibin\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysctl-d\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-systemd\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-node-log\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-cnibin\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-device-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvp8\" (UniqueName: \"kubernetes.io/projected/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-kube-api-access-rgvp8\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysctl-conf\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovnkube-script-lib\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-kubelet\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbg2\" (UniqueName: \"kubernetes.io/projected/cba5dada-eba5-494a-a80f-4ad39b08adc6-kube-api-access-hlbg2\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.844066 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21fc0a09-f140-4779-9002-78b4768a5e8e-tmp\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.844938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.843975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.844938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.844005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-sys\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.844938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.844055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.845215 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.845281 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-os-release\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.845281 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49hp\" (UniqueName: \"kubernetes.io/projected/42cf06e5-5e66-4b39-9671-d88da0a957db-kube-api-access-v49hp\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.845391 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.845391 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-kubernetes\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.845391 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68mn\" (UniqueName: \"kubernetes.io/projected/0c66c49e-4a7c-4a50-9015-1885258dd87a-kube-api-access-d68mn\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.845537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-multus-certs\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.845537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:46.845537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-tuned\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.845537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-env-overrides\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.845537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77cx\" (UniqueName: \"kubernetes.io/projected/aae37dc3-45bf-4c81-a4af-40683d4968ec-kube-api-access-v77cx\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.845866 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqdl\" (UniqueName: \"kubernetes.io/projected/21fc0a09-f140-4779-9002-78b4768a5e8e-kube-api-access-tfqdl\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.845866 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-os-release\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.845866 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.845609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-system-cni-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.848729 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.847267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gw6n9\""
Jul 28 23:21:46.848729 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.847614    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 23:21:46.848729 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.848213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 23:21:46.849075 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.848826    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 23:21:46.849390 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.849362    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 23:21:46.849846 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.849693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:46.849846 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.849815    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:21:46.851338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.850341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v2hlg\""
Jul 28 23:21:46.851338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.850391    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 23:21:46.851338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.850713    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 23:21:46.866936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.866908    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vqvzf"
Jul 28 23:21:46.874357 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.874331    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vqvzf"
Jul 28 23:21:46.933436 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.933358    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 23:21:46.945944 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.945906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-socket-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.946113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.945950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-sys-fs\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.946113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.945978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-iptables-alerter-script\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.946113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-host\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.946113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-host\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:46.946113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42cf06e5-5e66-4b39-9671-d88da0a957db-cni-binary-copy\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-conf-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-daemon-config\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-host-slash\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-etc-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87j4\" (UniqueName: \"kubernetes.io/projected/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-kube-api-access-g87j4\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-kubelet-config\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-k8s-cni-cncf-io\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-slash\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-cni-bin\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovnkube-config\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-tmp-dir\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-cni-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-cni-bin\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-systemd-units\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-ovn\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6fb9b58-ad38-4117-88c6-8465b0ba3496-konnectivity-ca\") pod \"konnectivity-agent-86l88\" (UID: \"d6fb9b58-ad38-4117-88c6-8465b0ba3496\") " pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-dbus\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-cni-multus\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-hostroot\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-kubelet\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-cni-netd\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-socket-dir-parent\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.946802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-registration-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-modprobe-d\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-lib-modules\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6fb9b58-ad38-4117-88c6-8465b0ba3496-agent-certs\") pod \"konnectivity-agent-86l88\" (UID: \"d6fb9b58-ad38-4117-88c6-8465b0ba3496\") " pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.946996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-cnibin\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysctl-d\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-systemd\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-node-log\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-cnibin\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-device-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvp8\" (UniqueName: \"kubernetes.io/projected/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-kube-api-access-rgvp8\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysctl-conf\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovnkube-script-lib\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-kubelet\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.947301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbg2\" (UniqueName: \"kubernetes.io/projected/cba5dada-eba5-494a-a80f-4ad39b08adc6-kube-api-access-hlbg2\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21fc0a09-f140-4779-9002-78b4768a5e8e-tmp\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-sys\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f8bn\" (UniqueName: \"kubernetes.io/projected/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-kube-api-access-6f8bn\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-os-release\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49hp\" (UniqueName: \"kubernetes.io/projected/42cf06e5-5e66-4b39-9671-d88da0a957db-kube-api-access-v49hp\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-kubernetes\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68mn\" (UniqueName: \"kubernetes.io/projected/0c66c49e-4a7c-4a50-9015-1885258dd87a-kube-api-access-d68mn\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-serviceca\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-multus-certs\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-tuned\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-env-overrides\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.948008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqvv\" (UniqueName: \"kubernetes.io/projected/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-kube-api-access-gkqvv\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77cx\" (UniqueName: \"kubernetes.io/projected/aae37dc3-45bf-4c81-a4af-40683d4968ec-kube-api-access-v77cx\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqdl\" (UniqueName: \"kubernetes.io/projected/21fc0a09-f140-4779-9002-78b4768a5e8e-kube-api-access-tfqdl\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-os-release\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-system-cni-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.947982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-systemd\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-run\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-system-cni-dir\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-netns\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-var-lib-kubelet\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-run-netns\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-var-lib-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-hosts-file\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysconfig\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-log-socket\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.948585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovn-node-metrics-cert\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.949237 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-etc-kubernetes\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.949237 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-etc-kubernetes\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.949237 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-socket-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.949237 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-sys-fs\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.949237 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-device-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.949237 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.948938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-host\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.949453 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.949309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-iptables-alerter-script\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.949512 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.949478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42cf06e5-5e66-4b39-9671-d88da0a957db-cni-binary-copy\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.949559 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.949540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-conf-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.949761 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.949737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysctl-conf\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.950064 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-daemon-config\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.950154 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-host-slash\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.950214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-etc-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.950267 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-k8s-cni-cncf-io\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.950325 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-slash\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.950325 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovnkube-script-lib\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.950431 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.950431 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-kubelet\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.950431 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-cni-bin\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.950628 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-cni-bin\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.950751 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.950820 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-registration-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.950877 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-systemd-units\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.950877 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950858    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 23:21:46.950984 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-ovn\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.950984 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-var-lib-cni-multus\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.950984 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-modprobe-d\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.950984 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-hostroot\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.951189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-cni-netd\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.951189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-lib-modules\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.951189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-socket-dir-parent\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.951189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-kubelet\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.951189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-systemd\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.951189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysctl-d\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.951491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-run-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.951491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-node-log\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.951491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-cnibin\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.951491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.950616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-multus-cni-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.951491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-cnibin\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.951491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cba5dada-eba5-494a-a80f-4ad39b08adc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.951806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-kubernetes\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.951806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.951806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-sys\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.951806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.952000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-multus-certs\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.952000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.951844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-host-run-netns\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.952106 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-var-lib-kubelet\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.952224 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.952224 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-host-run-netns\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.952224 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-var-lib-openvswitch\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.952224 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-run\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.952224 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-os-release\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.952430 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-sysconfig\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.952430 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-os-release\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.952430 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.952430 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42cf06e5-5e66-4b39-9671-d88da0a957db-system-cni-dir\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.952430 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aae37dc3-45bf-4c81-a4af-40683d4968ec-system-cni-dir\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.952430 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-systemd\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.952685 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c66c49e-4a7c-4a50-9015-1885258dd87a-log-socket\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.952809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovnkube-config\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.952809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c66c49e-4a7c-4a50-9015-1885258dd87a-env-overrides\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.953014 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.952852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aae37dc3-45bf-4c81-a4af-40683d4968ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.956055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.956031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21fc0a09-f140-4779-9002-78b4768a5e8e-tmp\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.956201 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.956180    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 23:21:46.956264 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.956208    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 23:21:46.956264 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.956222    2581 projected.go:196] Error preparing data for projected volume kube-api-access-thwvd for pod openshift-network-diagnostics/network-check-target-cbw8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:46.956340 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:46.956284    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd podName:5e6fa954-a3ac-48aa-97de-5d2b150c427a nodeName:}" failed. No retries permitted until 2026-07-28 23:21:47.456263308 +0000 UTC m=+2.958809843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-thwvd" (UniqueName: "kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd") pod "network-check-target-cbw8q" (UID: "5e6fa954-a3ac-48aa-97de-5d2b150c427a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:46.957092 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.956922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/21fc0a09-f140-4779-9002-78b4768a5e8e-etc-tuned\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.957572 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.957526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c66c49e-4a7c-4a50-9015-1885258dd87a-ovn-node-metrics-cert\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.960114 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.959998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvp8\" (UniqueName: \"kubernetes.io/projected/ab649bd7-fb71-4235-a3f1-0bf6105da3bf-kube-api-access-rgvp8\") pod \"iptables-alerter-9xtbg\" (UID: \"ab649bd7-fb71-4235-a3f1-0bf6105da3bf\") " pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:46.960114 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.960031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77cx\" (UniqueName: \"kubernetes.io/projected/aae37dc3-45bf-4c81-a4af-40683d4968ec-kube-api-access-v77cx\") pod \"multus-additional-cni-plugins-8xxts\" (UID: \"aae37dc3-45bf-4c81-a4af-40683d4968ec\") " pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:46.961861 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.961764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49hp\" (UniqueName: \"kubernetes.io/projected/42cf06e5-5e66-4b39-9671-d88da0a957db-kube-api-access-v49hp\") pod \"multus-h9td4\" (UID: \"42cf06e5-5e66-4b39-9671-d88da0a957db\") " pod="openshift-multus/multus-h9td4"
Jul 28 23:21:46.961861 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.961802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68mn\" (UniqueName: \"kubernetes.io/projected/0c66c49e-4a7c-4a50-9015-1885258dd87a-kube-api-access-d68mn\") pod \"ovnkube-node-ms6kt\" (UID: \"0c66c49e-4a7c-4a50-9015-1885258dd87a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:46.962472 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.962449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbg2\" (UniqueName: \"kubernetes.io/projected/cba5dada-eba5-494a-a80f-4ad39b08adc6-kube-api-access-hlbg2\") pod \"aws-ebs-csi-driver-node-zcpnk\" (UID: \"cba5dada-eba5-494a-a80f-4ad39b08adc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:46.963657 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.963611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqdl\" (UniqueName: \"kubernetes.io/projected/21fc0a09-f140-4779-9002-78b4768a5e8e-kube-api-access-tfqdl\") pod \"tuned-wd8jd\" (UID: \"21fc0a09-f140-4779-9002-78b4768a5e8e\") " pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:46.980909 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.980857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" event={"ID":"e097d8f7f5bef6b1daa1f2fc30e6048e","Type":"ContainerStarted","Data":"42c24685d4d1a5d350807993feb6b157e35c1c77f3a268cf22fa9afc4008c2e7"}
Jul 28 23:21:46.982028 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:46.982002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" event={"ID":"a0594a384a49868ef7a8441f356bc9f2","Type":"ContainerStarted","Data":"55a1a19c04a0aaaae1e590e677e5f319e0bd5433736e426b4f5fa6a541aa079b"}
Jul 28 23:21:47.048797 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6fb9b58-ad38-4117-88c6-8465b0ba3496-agent-certs\") pod \"konnectivity-agent-86l88\" (UID: \"d6fb9b58-ad38-4117-88c6-8465b0ba3496\") " pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:47.048797 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f8bn\" (UniqueName: \"kubernetes.io/projected/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-kube-api-access-6f8bn\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:47.049014 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:47.049014 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-serviceca\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:47.049014 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqvv\" (UniqueName: \"kubernetes.io/projected/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-kube-api-access-gkqvv\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:47.049014 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:47.049014 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-hosts-file\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:47.049014 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.048999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-host\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:47.049292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87j4\" (UniqueName: \"kubernetes.io/projected/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-kube-api-access-g87j4\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:47.049292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-kubelet-config\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:47.049292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-tmp-dir\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:47.049292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6fb9b58-ad38-4117-88c6-8465b0ba3496-konnectivity-ca\") pod \"konnectivity-agent-86l88\" (UID: \"d6fb9b58-ad38-4117-88c6-8465b0ba3496\") " pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:47.049292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-dbus\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:47.049292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-dbus\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:47.049647 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.049597    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:47.049728 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.049690    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret podName:17ed82b0-64ac-4fe9-aff4-3ef02f1476a0 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:47.549655615 +0000 UTC m=+3.052202143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret") pod "global-pull-secret-syncer-fkq46" (UID: "17ed82b0-64ac-4fe9-aff4-3ef02f1476a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:47.049875 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-host\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:47.049984 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.049963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-hosts-file\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:47.050032 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.049997    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:47.050097 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.050068    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:47.550049899 +0000 UTC m=+3.052596421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:47.050157 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.050093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-kubelet-config\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:47.050219 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.050155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-tmp-dir\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:47.050478 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.050459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-serviceca\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:47.050645 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.050599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6fb9b58-ad38-4117-88c6-8465b0ba3496-konnectivity-ca\") pod \"konnectivity-agent-86l88\" (UID: \"d6fb9b58-ad38-4117-88c6-8465b0ba3496\") " pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:47.052769 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.052746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6fb9b58-ad38-4117-88c6-8465b0ba3496-agent-certs\") pod \"konnectivity-agent-86l88\" (UID: \"d6fb9b58-ad38-4117-88c6-8465b0ba3496\") " pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:47.058318 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.058274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f8bn\" (UniqueName: \"kubernetes.io/projected/9125b4bf-4b57-4adb-a1c6-9eb94d4525db-kube-api-access-6f8bn\") pod \"node-resolver-w5rtn\" (UID: \"9125b4bf-4b57-4adb-a1c6-9eb94d4525db\") " pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:47.058877 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.058854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87j4\" (UniqueName: \"kubernetes.io/projected/0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91-kube-api-access-g87j4\") pod \"node-ca-jvngb\" (UID: \"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91\") " pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:47.059047 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.059022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqvv\" (UniqueName: \"kubernetes.io/projected/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-kube-api-access-gkqvv\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:47.134004 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.133963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:21:47.142254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.142228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xxts"
Jul 28 23:21:47.155342 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.155318    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h9td4"
Jul 28 23:21:47.163378 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.163359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk"
Jul 28 23:21:47.174165 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.174141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9xtbg"
Jul 28 23:21:47.181024 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.181001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wd8jd"
Jul 28 23:21:47.190864 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.190844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:21:47.199569 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.199547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w5rtn"
Jul 28 23:21:47.207177 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.207156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvngb"
Jul 28 23:21:47.365944 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.365866    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 23:21:47.553976 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.553935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:47.554148 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.553994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:47.554148 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.554040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:47.554148 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554126    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 23:21:47.554321 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554155    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 23:21:47.554321 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554160    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:47.554321 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554168    2581 projected.go:196] Error preparing data for projected volume kube-api-access-thwvd for pod openshift-network-diagnostics/network-check-target-cbw8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:47.554321 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554161    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:47.554321 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554222    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret podName:17ed82b0-64ac-4fe9-aff4-3ef02f1476a0 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:48.554201302 +0000 UTC m=+4.056747815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret") pod "global-pull-secret-syncer-fkq46" (UID: "17ed82b0-64ac-4fe9-aff4-3ef02f1476a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:47.554321 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554241    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd podName:5e6fa954-a3ac-48aa-97de-5d2b150c427a nodeName:}" failed. No retries permitted until 2026-07-28 23:21:48.554231762 +0000 UTC m=+4.056778279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-thwvd" (UniqueName: "kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd") pod "network-check-target-cbw8q" (UID: "5e6fa954-a3ac-48aa-97de-5d2b150c427a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:47.554321 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.554260    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:48.554250682 +0000 UTC m=+4.056797193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:47.878258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.878187    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 23:16:47 +0000 UTC" deadline="2028-01-01 21:53:13.765318728 +0000 UTC"
Jul 28 23:21:47.878258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.878245    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12526h31m25.887078605s"
Jul 28 23:21:47.975420 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:47.975383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:47.975611 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:47.975555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:21:48.072584 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.072553    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab649bd7_fb71_4235_a3f1_0bf6105da3bf.slice/crio-ff44cbf63f0fe6cc9c4b333624a4be3877a51b05abe46a63945a14cdc56104f1 WatchSource:0}: Error finding container ff44cbf63f0fe6cc9c4b333624a4be3877a51b05abe46a63945a14cdc56104f1: Status 404 returned error can't find the container with id ff44cbf63f0fe6cc9c4b333624a4be3877a51b05abe46a63945a14cdc56104f1
Jul 28 23:21:48.074004 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.073968    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae37dc3_45bf_4c81_a4af_40683d4968ec.slice/crio-abaadeee40a203ff618b3e83c2977cbcc6c9c2b678343be25018138a081b79aa WatchSource:0}: Error finding container abaadeee40a203ff618b3e83c2977cbcc6c9c2b678343be25018138a081b79aa: Status 404 returned error can't find the container with id abaadeee40a203ff618b3e83c2977cbcc6c9c2b678343be25018138a081b79aa
Jul 28 23:21:48.074979 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.074896    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c66c49e_4a7c_4a50_9015_1885258dd87a.slice/crio-995a218630b2c8b643fd46531031c1c30309d7e7f44462aa2a91512435b7b5c6 WatchSource:0}: Error finding container 995a218630b2c8b643fd46531031c1c30309d7e7f44462aa2a91512435b7b5c6: Status 404 returned error can't find the container with id 995a218630b2c8b643fd46531031c1c30309d7e7f44462aa2a91512435b7b5c6
Jul 28 23:21:48.077270 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.076895    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9125b4bf_4b57_4adb_a1c6_9eb94d4525db.slice/crio-d121d6122c7130dac80fe95cedd67e6a93195099b99bb972a9d60d1a6afbc560 WatchSource:0}: Error finding container d121d6122c7130dac80fe95cedd67e6a93195099b99bb972a9d60d1a6afbc560: Status 404 returned error can't find the container with id d121d6122c7130dac80fe95cedd67e6a93195099b99bb972a9d60d1a6afbc560
Jul 28 23:21:48.077270 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.077155    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cf06e5_5e66_4b39_9671_d88da0a957db.slice/crio-d5ced62aca54a0867c27c2d27024a972234672cd1810ed44123fe03ca1d08c7b WatchSource:0}: Error finding container d5ced62aca54a0867c27c2d27024a972234672cd1810ed44123fe03ca1d08c7b: Status 404 returned error can't find the container with id d5ced62aca54a0867c27c2d27024a972234672cd1810ed44123fe03ca1d08c7b
Jul 28 23:21:48.102270 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.102237    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae77f6d_9b0e_4784_8f9d_1f89fbdfdd91.slice/crio-739d89783d8ee2c77cb1280d23cb581eeffac2505fc7b91a04a923453476c8f0 WatchSource:0}: Error finding container 739d89783d8ee2c77cb1280d23cb581eeffac2505fc7b91a04a923453476c8f0: Status 404 returned error can't find the container with id 739d89783d8ee2c77cb1280d23cb581eeffac2505fc7b91a04a923453476c8f0
Jul 28 23:21:48.103614 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.103584    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba5dada_eba5_494a_a80f_4ad39b08adc6.slice/crio-b0ec39ebf5f54ace4f872e3289cf48ed53d8e074b0dba40a3c0db3f93f92756f WatchSource:0}: Error finding container b0ec39ebf5f54ace4f872e3289cf48ed53d8e074b0dba40a3c0db3f93f92756f: Status 404 returned error can't find the container with id b0ec39ebf5f54ace4f872e3289cf48ed53d8e074b0dba40a3c0db3f93f92756f
Jul 28 23:21:48.104378 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.104354    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fb9b58_ad38_4117_88c6_8465b0ba3496.slice/crio-ecc7341bd770750ae36c7bdeaf6496e24ffcc396f8b7c8f8952d29f79a6dd20d WatchSource:0}: Error finding container ecc7341bd770750ae36c7bdeaf6496e24ffcc396f8b7c8f8952d29f79a6dd20d: Status 404 returned error can't find the container with id ecc7341bd770750ae36c7bdeaf6496e24ffcc396f8b7c8f8952d29f79a6dd20d
Jul 28 23:21:48.106165 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:21:48.106117    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fc0a09_f140_4779_9002_78b4768a5e8e.slice/crio-e9d496a623651dedf1db7349ba16369799cced75d070941d979d7c672f14aa80 WatchSource:0}: Error finding container e9d496a623651dedf1db7349ba16369799cced75d070941d979d7c672f14aa80: Status 404 returned error can't find the container with id e9d496a623651dedf1db7349ba16369799cced75d070941d979d7c672f14aa80
Jul 28 23:21:48.563713 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:48.563381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:48.563713 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.563516    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:48.563713 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:48.563700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:48.563953 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.563753    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret podName:17ed82b0-64ac-4fe9-aff4-3ef02f1476a0 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:50.563728909 +0000 UTC m=+6.066275437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret") pod "global-pull-secret-syncer-fkq46" (UID: "17ed82b0-64ac-4fe9-aff4-3ef02f1476a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:48.563953 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.563813    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:48.563953 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:48.563818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:48.563953 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.563870    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:50.563849804 +0000 UTC m=+6.066396318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:48.564181 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.563956    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 23:21:48.564181 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.563975    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 23:21:48.564181 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.563989    2581 projected.go:196] Error preparing data for projected volume kube-api-access-thwvd for pod openshift-network-diagnostics/network-check-target-cbw8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:48.564181 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.564029    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd podName:5e6fa954-a3ac-48aa-97de-5d2b150c427a nodeName:}" failed. No retries permitted until 2026-07-28 23:21:50.564014822 +0000 UTC m=+6.066561337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-thwvd" (UniqueName: "kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd") pod "network-check-target-cbw8q" (UID: "5e6fa954-a3ac-48aa-97de-5d2b150c427a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:48.878436 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:48.878337    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 23:16:47 +0000 UTC" deadline="2028-02-07 05:35:32.578886741 +0000 UTC"
Jul 28 23:21:48.878436 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:48.878383    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13398h13m43.700508266s"
Jul 28 23:21:48.977418 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:48.976854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:48.977418 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.977009    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:21:48.977934 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:48.977774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:48.977934 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:48.977890    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:21:49.004951 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.004918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" event={"ID":"a0594a384a49868ef7a8441f356bc9f2","Type":"ContainerStarted","Data":"3cf7e6668ccc9f7845821d8f7210f5e38b7a0ab8a0cd4dd76c8754409df49235"}
Jul 28 23:21:49.012907 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.012815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk" event={"ID":"cba5dada-eba5-494a-a80f-4ad39b08adc6","Type":"ContainerStarted","Data":"b0ec39ebf5f54ace4f872e3289cf48ed53d8e074b0dba40a3c0db3f93f92756f"}
Jul 28 23:21:49.018200 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.018167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"995a218630b2c8b643fd46531031c1c30309d7e7f44462aa2a91512435b7b5c6"}
Jul 28 23:21:49.027875 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.027823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerStarted","Data":"abaadeee40a203ff618b3e83c2977cbcc6c9c2b678343be25018138a081b79aa"}
Jul 28 23:21:49.035356 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.034442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9xtbg" event={"ID":"ab649bd7-fb71-4235-a3f1-0bf6105da3bf","Type":"ContainerStarted","Data":"ff44cbf63f0fe6cc9c4b333624a4be3877a51b05abe46a63945a14cdc56104f1"}
Jul 28 23:21:49.039623 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.039554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wd8jd" event={"ID":"21fc0a09-f140-4779-9002-78b4768a5e8e","Type":"ContainerStarted","Data":"e9d496a623651dedf1db7349ba16369799cced75d070941d979d7c672f14aa80"}
Jul 28 23:21:49.050587 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.050560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-86l88" event={"ID":"d6fb9b58-ad38-4117-88c6-8465b0ba3496","Type":"ContainerStarted","Data":"ecc7341bd770750ae36c7bdeaf6496e24ffcc396f8b7c8f8952d29f79a6dd20d"}
Jul 28 23:21:49.055406 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.055311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvngb" event={"ID":"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91","Type":"ContainerStarted","Data":"739d89783d8ee2c77cb1280d23cb581eeffac2505fc7b91a04a923453476c8f0"}
Jul 28 23:21:49.058411 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.058324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h9td4" event={"ID":"42cf06e5-5e66-4b39-9671-d88da0a957db","Type":"ContainerStarted","Data":"d5ced62aca54a0867c27c2d27024a972234672cd1810ed44123fe03ca1d08c7b"}
Jul 28 23:21:49.066159 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.066115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w5rtn" event={"ID":"9125b4bf-4b57-4adb-a1c6-9eb94d4525db","Type":"ContainerStarted","Data":"d121d6122c7130dac80fe95cedd67e6a93195099b99bb972a9d60d1a6afbc560"}
Jul 28 23:21:49.091566 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.090947    2581 generic.go:356] "Generic (PLEG): container finished" podID="e097d8f7f5bef6b1daa1f2fc30e6048e" containerID="3db1001786a95fc239edd16a72c4b34759420a3ad82dde200733fd02f1b42952" exitCode=0
Jul 28 23:21:49.091566 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.090983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" event={"ID":"e097d8f7f5bef6b1daa1f2fc30e6048e","Type":"ContainerDied","Data":"3db1001786a95fc239edd16a72c4b34759420a3ad82dde200733fd02f1b42952"}
Jul 28 23:21:49.109633 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.109582    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-158.ec2.internal" podStartSLOduration=3.109567676 podStartE2EDuration="3.109567676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:21:49.020202897 +0000 UTC m=+4.522749433" watchObservedRunningTime="2026-07-28 23:21:49.109567676 +0000 UTC m=+4.612114210"
Jul 28 23:21:49.977179 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:49.976339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:49.977179 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:49.976469    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:21:50.127285 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:50.127184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" event={"ID":"e097d8f7f5bef6b1daa1f2fc30e6048e","Type":"ContainerStarted","Data":"ca9d27b9cf0d13a09a5d163bad18a2976647f3bf70c2facf19834c8b42e27cd4"}
Jul 28 23:21:50.144978 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:50.144910    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-158.ec2.internal" podStartSLOduration=4.144887667 podStartE2EDuration="4.144887667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:21:50.144769975 +0000 UTC m=+5.647316514" watchObservedRunningTime="2026-07-28 23:21:50.144887667 +0000 UTC m=+5.647434198"
Jul 28 23:21:50.583811 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:50.583773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:50.583999 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:50.583836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:50.583999 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:50.583881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:50.584111 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584012    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:50.584111 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584076    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:54.584056966 +0000 UTC m=+10.086603479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:50.584508 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584489    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 23:21:50.584610 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584516    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 23:21:50.584610 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584529    2581 projected.go:196] Error preparing data for projected volume kube-api-access-thwvd for pod openshift-network-diagnostics/network-check-target-cbw8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:50.584610 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584576    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd podName:5e6fa954-a3ac-48aa-97de-5d2b150c427a nodeName:}" failed. No retries permitted until 2026-07-28 23:21:54.584560337 +0000 UTC m=+10.087106854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-thwvd" (UniqueName: "kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd") pod "network-check-target-cbw8q" (UID: "5e6fa954-a3ac-48aa-97de-5d2b150c427a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:50.584839 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584650    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:50.584839 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.584682    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret podName:17ed82b0-64ac-4fe9-aff4-3ef02f1476a0 nodeName:}" failed. No retries permitted until 2026-07-28 23:21:54.584671697 +0000 UTC m=+10.087218210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret") pod "global-pull-secret-syncer-fkq46" (UID: "17ed82b0-64ac-4fe9-aff4-3ef02f1476a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:50.976832 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:50.976736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:50.976832 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:50.976748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:50.977034 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.976873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:21:50.977034 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:50.976954    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:21:51.976288 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:51.976252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:51.976763 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:51.976374    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:21:52.980801 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:52.980121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:52.980801 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:52.980242    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:21:52.980801 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:52.980611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:52.980801 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:52.980747    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:21:53.976141 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:53.975828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:53.976141 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:53.975955    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:21:54.619958 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:54.619917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:54.620426 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:54.619987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:54.620426 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:54.620022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:54.620426 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620141    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:54.620426 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620198    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret podName:17ed82b0-64ac-4fe9-aff4-3ef02f1476a0 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:02.620179829 +0000 UTC m=+18.122726352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret") pod "global-pull-secret-syncer-fkq46" (UID: "17ed82b0-64ac-4fe9-aff4-3ef02f1476a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 23:21:54.620668 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620590    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:54.620732 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620668    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:02.620650534 +0000 UTC m=+18.123197045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:21:54.620791 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620752    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 23:21:54.620791 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620768    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 23:21:54.620791 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620780    2581 projected.go:196] Error preparing data for projected volume kube-api-access-thwvd for pod openshift-network-diagnostics/network-check-target-cbw8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:54.620931 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.620817    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd podName:5e6fa954-a3ac-48aa-97de-5d2b150c427a nodeName:}" failed. No retries permitted until 2026-07-28 23:22:02.62080545 +0000 UTC m=+18.123351982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-thwvd" (UniqueName: "kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd") pod "network-check-target-cbw8q" (UID: "5e6fa954-a3ac-48aa-97de-5d2b150c427a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:21:54.978136 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:54.976725    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:54.978136 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.976864    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:21:54.978333 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:54.978163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:54.978333 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:54.978255    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:21:55.975488 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:55.975453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:55.975963 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:55.975580    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:21:56.976008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:56.975972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:56.976482 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:56.976016    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:56.976482 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:56.976106    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:21:56.976608 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:56.976510    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:21:57.975695 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:57.975653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:57.975874 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:57.975810    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:21:58.976245 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:58.976152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:21:58.976744 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:58.976284    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:21:58.976744 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:58.976153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:21:58.976744 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:58.976672    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:21:59.976124 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:21:59.976084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:21:59.976325 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:21:59.976207    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:00.976044 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:00.975997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:00.976224 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:00.976136    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:00.976318 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:00.976001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:00.976697 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:00.976415    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:01.975803 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:01.975761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:01.976049 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:01.975874    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:02.676515 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:02.676420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:02.676515 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:02.676477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:02.676515 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:02.676512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676602    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676617    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676639    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676652    2581 projected.go:196] Error preparing data for projected volume kube-api-access-thwvd for pod openshift-network-diagnostics/network-check-target-cbw8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676689    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:18.67666858 +0000 UTC m=+34.179215094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676708    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd podName:5e6fa954-a3ac-48aa-97de-5d2b150c427a nodeName:}" failed. No retries permitted until 2026-07-28 23:22:18.676698715 +0000 UTC m=+34.179245224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-thwvd" (UniqueName: "kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd") pod "network-check-target-cbw8q" (UID: "5e6fa954-a3ac-48aa-97de-5d2b150c427a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676723    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 23:22:02.677084 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.676770    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret podName:17ed82b0-64ac-4fe9-aff4-3ef02f1476a0 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:18.676753823 +0000 UTC m=+34.179300345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret") pod "global-pull-secret-syncer-fkq46" (UID: "17ed82b0-64ac-4fe9-aff4-3ef02f1476a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 23:22:02.975670 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:02.975565    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:02.975880 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.975725    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:02.975880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:02.975782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:02.976006 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:02.975899    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:03.975313 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:03.975275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:03.975774 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:03.975405    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:04.976022 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:04.975989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:04.976471 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:04.976086    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:04.976471 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:04.976171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:04.976471 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:04.976291    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:05.975680 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:05.975645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:05.975862 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:05.975812    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:06.976246 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:06.975925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:06.976901 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:06.975973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:06.976901 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:06.976349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:06.976901 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:06.976408    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:07.163439 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.163403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w5rtn" event={"ID":"9125b4bf-4b57-4adb-a1c6-9eb94d4525db","Type":"ContainerStarted","Data":"d33a6ee8ef0f38ab6c3824a70486ceb21bf7940c3a13915b14cbc9db0fd715f0"}
Jul 28 23:22:07.165136 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.165108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk" event={"ID":"cba5dada-eba5-494a-a80f-4ad39b08adc6","Type":"ContainerStarted","Data":"7d8af7e1803226e53c6bb7439aa6f35b1d17bf962b471be90942f2432708171d"}
Jul 28 23:22:07.168487 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.168460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"2d6536f5de58820134bb4796bf469ecc3e31f828b132a67d76deb2db1cb77297"}
Jul 28 23:22:07.168600 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.168544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"6fca716cba37aecb76b272ea826edcbe9d6599c1721dfc7ef64806c516f8d86d"}
Jul 28 23:22:07.168600 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.168592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"a39379d36b2d5ef743d5a710454e758575d964d31f3c6195127731292a242f43"}
Jul 28 23:22:07.168712 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.168662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"9438db43714a3aa6ca7e05b03f5c78855f7b060727047d19c53d5ead967ef435"}
Jul 28 23:22:07.168712 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.168709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"183edaaf618f6e9f30882a1a7d70beb21bc95f43ff00842362424496894167ba"}
Jul 28 23:22:07.168821 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.168757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"6594fbc7613a9dfd2fea4c5bae68a2391f5c072f87dd8efaf95f8ab552583334"}
Jul 28 23:22:07.170009 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.169973    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae37dc3-45bf-4c81-a4af-40683d4968ec" containerID="44e083b9f65077aa576afa1781db03b8e213b81e80bbc6494b7fc754b2c534ea" exitCode=0
Jul 28 23:22:07.170124 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.170062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerDied","Data":"44e083b9f65077aa576afa1781db03b8e213b81e80bbc6494b7fc754b2c534ea"}
Jul 28 23:22:07.171826 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.171746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wd8jd" event={"ID":"21fc0a09-f140-4779-9002-78b4768a5e8e","Type":"ContainerStarted","Data":"12a45ca2bd22ef320e08c3548d2f5982286b0585041fe98a50d587f41d7c1c23"}
Jul 28 23:22:07.173555 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.173531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-86l88" event={"ID":"d6fb9b58-ad38-4117-88c6-8465b0ba3496","Type":"ContainerStarted","Data":"2b7254965faec4a588fb528bd31722322678ece06d59440e556ec2fa9a0b6465"}
Jul 28 23:22:07.177897 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.177865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvngb" event={"ID":"0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91","Type":"ContainerStarted","Data":"30d8e1e57c41364de14dc181cd20f5e722315a90b372eeedffba2b1b89b206ea"}
Jul 28 23:22:07.178024 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.177913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w5rtn" podStartSLOduration=3.827779253 podStartE2EDuration="22.177895781s" totalImagesPullingTime="18.350116528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:07.177251622 +0000 UTC m=+22.679798154" watchObservedRunningTime="2026-07-28 23:22:07.177895781 +0000 UTC m=+22.680442312"
Jul 28 23:22:07.179789 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.179765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h9td4" event={"ID":"42cf06e5-5e66-4b39-9671-d88da0a957db","Type":"ContainerStarted","Data":"0f45d962262d7cb806c3a2ca618a13ec6c43eaea026eb75bba8caf77941f9806"}
Jul 28 23:22:07.190794 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.190715    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-86l88" podStartSLOduration=7.958784195 podStartE2EDuration="22.190697407s" totalImagesPullingTime="14.231913212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:07.190566382 +0000 UTC m=+22.693112913" watchObservedRunningTime="2026-07-28 23:22:07.190697407 +0000 UTC m=+22.693243938"
Jul 28 23:22:07.222983 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.222935    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wd8jd" podStartSLOduration=3.879053268 podStartE2EDuration="22.222920488s" totalImagesPullingTime="18.34386722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:07.222814431 +0000 UTC m=+22.725360964" watchObservedRunningTime="2026-07-28 23:22:07.222920488 +0000 UTC m=+22.725467019"
Jul 28 23:22:07.237748 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.237706    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h9td4" podStartSLOduration=3.842398329 podStartE2EDuration="22.237690792s" totalImagesPullingTime="18.395292463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:07.237247269 +0000 UTC m=+22.739793799" watchObservedRunningTime="2026-07-28 23:22:07.237690792 +0000 UTC m=+22.740237323"
Jul 28 23:22:07.254286 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.254234    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jvngb" podStartSLOduration=8.022022176 podStartE2EDuration="22.254216426s" totalImagesPullingTime="14.23219425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:07.253864311 +0000 UTC m=+22.756410841" watchObservedRunningTime="2026-07-28 23:22:07.254216426 +0000 UTC m=+22.756762959"
Jul 28 23:22:07.337643 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.337590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:22:07.338363 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.338343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:22:07.976152 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:07.976109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:07.976329 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:07.976247    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:08.075085 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.075063    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 23:22:08.184198 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.184109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk" event={"ID":"cba5dada-eba5-494a-a80f-4ad39b08adc6","Type":"ContainerStarted","Data":"454d3eea1fb63699b7c6b6dec2c72f4c2dcde08c0363ea3e29dca57506f0ca1c"}
Jul 28 23:22:08.186467 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.186436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9xtbg" event={"ID":"ab649bd7-fb71-4235-a3f1-0bf6105da3bf","Type":"ContainerStarted","Data":"8f0d0bbc9fc02d551dbba4900fd2f3e42b4ccea39d7aecad0102a1706da8d572"}
Jul 28 23:22:08.187240 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.187219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:22:08.187365 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.187305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-86l88"
Jul 28 23:22:08.201261 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.201216    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9xtbg" podStartSLOduration=8.937165027 podStartE2EDuration="23.201201348s" totalImagesPullingTime="14.264036321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:08.201085839 +0000 UTC m=+23.703632371" watchObservedRunningTime="2026-07-28 23:22:08.201201348 +0000 UTC m=+23.703747878"
Jul 28 23:22:08.895275 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.895070    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T23:22:08.075078951Z","UUID":"04b84493-c561-4df3-85d0-38c34c376498","Handler":null,"Name":"","Endpoint":""}
Jul 28 23:22:08.896932 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.896768    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 23:22:08.896932 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.896811    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 23:22:08.975604 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.975576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:08.975792 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:08.975576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:08.975792 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:08.975707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:08.975905 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:08.975786    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:09.192360 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:09.192275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"a123630a6d3700ebd2dd705017aa0cf55c79526c65a517260edcd86e3d0fa9a6"}
Jul 28 23:22:09.976018 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:09.975988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:09.976176 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:09.976085    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:10.198482 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:10.198293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk" event={"ID":"cba5dada-eba5-494a-a80f-4ad39b08adc6","Type":"ContainerStarted","Data":"9d75f0c3be4f90e9e323b3d999d42f32941c977f853e06e30459b6f069a98493"}
Jul 28 23:22:10.215140 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:10.215082    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zcpnk" podStartSLOduration=4.344005866 podStartE2EDuration="25.215067873s" totalImagesPullingTime="20.871062007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:10.214953977 +0000 UTC m=+25.717500513" watchObservedRunningTime="2026-07-28 23:22:10.215067873 +0000 UTC m=+25.717614403"
Jul 28 23:22:10.976221 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:10.976187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:10.976424 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:10.976190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:10.976424 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:10.976332    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:10.976523 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:10.976414    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:11.976362 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:11.976328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:11.976936 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:11.976438    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:12.205641 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.205547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" event={"ID":"0c66c49e-4a7c-4a50-9015-1885258dd87a","Type":"ContainerStarted","Data":"2fc3adfc1e42adcd29440b23535dfe5494bf6d3ed5d397b844cfb85f9e2d9e2b"}
Jul 28 23:22:12.205937 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.205841    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:22:12.207434 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.207411    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae37dc3-45bf-4c81-a4af-40683d4968ec" containerID="092f9b29dda16bfa8e3bd12c68239878d3603386e2273ea676047831c7b6f4f3" exitCode=0
Jul 28 23:22:12.207537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.207446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerDied","Data":"092f9b29dda16bfa8e3bd12c68239878d3603386e2273ea676047831c7b6f4f3"}
Jul 28 23:22:12.222049 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.222027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:22:12.232804 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.232764    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt" podStartSLOduration=8.817641879 podStartE2EDuration="27.232750659s" totalImagesPullingTime="18.41510878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:12.232620718 +0000 UTC m=+27.735167249" watchObservedRunningTime="2026-07-28 23:22:12.232750659 +0000 UTC m=+27.735297204"
Jul 28 23:22:12.976282 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.976238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:12.976520 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:12.976393    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:12.976520 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:12.976451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:12.976918 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:12.976572    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:13.215277 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.215239    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae37dc3-45bf-4c81-a4af-40683d4968ec" containerID="c09381073e2d4f2518a1b0ac8e60076dc5ee6909438f0639e5ca20494542cd16" exitCode=0
Jul 28 23:22:13.215440 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.215318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerDied","Data":"c09381073e2d4f2518a1b0ac8e60076dc5ee6909438f0639e5ca20494542cd16"}
Jul 28 23:22:13.215731 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.215713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:22:13.215803 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.215784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:22:13.231577 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.231522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:22:13.801869 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.801834    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbw8q"]
Jul 28 23:22:13.802019 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.801951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:13.802062 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:13.802037    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:13.805370 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.805339    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m968q"]
Jul 28 23:22:13.805527 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.805463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:13.805609 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:13.805555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:13.807652 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.806739    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fkq46"]
Jul 28 23:22:13.807652 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:13.806997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:13.807652 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:13.807204    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:14.978749 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:14.978544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:14.979145 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:14.978609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:14.979145 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:14.978832    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:14.979145 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:14.978908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:15.222920 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:15.222890    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae37dc3-45bf-4c81-a4af-40683d4968ec" containerID="27bfa91d04fd568b8d8e8da579c5ba206f80cee589e465cfa4729e1551faabcc" exitCode=0
Jul 28 23:22:15.223068 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:15.222973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerDied","Data":"27bfa91d04fd568b8d8e8da579c5ba206f80cee589e465cfa4729e1551faabcc"}
Jul 28 23:22:15.976373 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:15.976336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:15.976548 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:15.976466    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:16.975922 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:16.975890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:16.976541 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:16.976019    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:16.976541 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:16.976084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:16.976541 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:16.976185    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:17.976246 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:17.976198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:17.976791 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:17.976335    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m968q" podUID="0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6"
Jul 28 23:22:18.695941 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:18.695906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:18.696124 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:18.695963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:18.696124 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:18.696004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:18.696124 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696060    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:22:18.696124 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696118    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:50.696099993 +0000 UTC m=+66.198646505 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 23:22:18.696330 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696162    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 23:22:18.696330 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696191    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 23:22:18.696330 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696195    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 23:22:18.696330 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696259    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret podName:17ed82b0-64ac-4fe9-aff4-3ef02f1476a0 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:50.69624062 +0000 UTC m=+66.198787132 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret") pod "global-pull-secret-syncer-fkq46" (UID: "17ed82b0-64ac-4fe9-aff4-3ef02f1476a0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 23:22:18.696330 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696201    2581 projected.go:196] Error preparing data for projected volume kube-api-access-thwvd for pod openshift-network-diagnostics/network-check-target-cbw8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:22:18.696330 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.696315    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd podName:5e6fa954-a3ac-48aa-97de-5d2b150c427a nodeName:}" failed. No retries permitted until 2026-07-28 23:22:50.696304606 +0000 UTC m=+66.198851117 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-thwvd" (UniqueName: "kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd") pod "network-check-target-cbw8q" (UID: "5e6fa954-a3ac-48aa-97de-5d2b150c427a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 23:22:18.976079 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:18.976048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:18.976240 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.976180    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cbw8q" podUID="5e6fa954-a3ac-48aa-97de-5d2b150c427a"
Jul 28 23:22:18.976240 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:18.976203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:18.976656 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:18.976286    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkq46" podUID="17ed82b0-64ac-4fe9-aff4-3ef02f1476a0"
Jul 28 23:22:19.235841 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.235809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerStarted","Data":"214beaa798e119bf6bcd121ad3ea7e060b768a5c2bb4202f329c61598343100e"}
Jul 28 23:22:19.315473 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.315447    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-158.ec2.internal" event="NodeReady"
Jul 28 23:22:19.315610 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.315598    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 23:22:19.355418 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.355389    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5c9l5"]
Jul 28 23:22:19.358263 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.358241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kbzfw"]
Jul 28 23:22:19.358382 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.358351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.360789 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.360758    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 23:22:19.360907 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.360808    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 23:22:19.360907 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.360854    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7kmkd\""
Jul 28 23:22:19.360907 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.360877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.365042 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.365025    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 23:22:19.365458 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.365436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 23:22:19.365551 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.365514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dpgd4\""
Jul 28 23:22:19.365551 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.365524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 23:22:19.367167 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.367148    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c9l5"]
Jul 28 23:22:19.375717 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.375695    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kbzfw"]
Jul 28 23:22:19.501905 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.501814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fsk\" (UniqueName: \"kubernetes.io/projected/e343f41c-bdac-477f-94b8-2b41ba74100d-kube-api-access-g4fsk\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.501905 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.501864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8q8\" (UniqueName: \"kubernetes.io/projected/4daeb74a-6c58-4c2b-836f-73d705ad0e5b-kube-api-access-cx8q8\") pod \"ingress-canary-kbzfw\" (UID: \"4daeb74a-6c58-4c2b-836f-73d705ad0e5b\") " pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.501905 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.501913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e343f41c-bdac-477f-94b8-2b41ba74100d-tmp-dir\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.502118 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.501939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e343f41c-bdac-477f-94b8-2b41ba74100d-config-volume\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.502118 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.501965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.502118 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.502016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4daeb74a-6c58-4c2b-836f-73d705ad0e5b-cert\") pod \"ingress-canary-kbzfw\" (UID: \"4daeb74a-6c58-4c2b-836f-73d705ad0e5b\") " pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.602415 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.602342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4daeb74a-6c58-4c2b-836f-73d705ad0e5b-cert\") pod \"ingress-canary-kbzfw\" (UID: \"4daeb74a-6c58-4c2b-836f-73d705ad0e5b\") " pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.602578 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.602442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fsk\" (UniqueName: \"kubernetes.io/projected/e343f41c-bdac-477f-94b8-2b41ba74100d-kube-api-access-g4fsk\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.602578 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.602460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8q8\" (UniqueName: \"kubernetes.io/projected/4daeb74a-6c58-4c2b-836f-73d705ad0e5b-kube-api-access-cx8q8\") pod \"ingress-canary-kbzfw\" (UID: \"4daeb74a-6c58-4c2b-836f-73d705ad0e5b\") " pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.602578 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.602490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e343f41c-bdac-477f-94b8-2b41ba74100d-tmp-dir\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.602578 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.602516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e343f41c-bdac-477f-94b8-2b41ba74100d-config-volume\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.602578 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.602537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.602871 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:19.602660    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 23:22:19.602871 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:19.602735    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls podName:e343f41c-bdac-477f-94b8-2b41ba74100d nodeName:}" failed. No retries permitted until 2026-07-28 23:22:20.102715605 +0000 UTC m=+35.605262129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls") pod "dns-default-5c9l5" (UID: "e343f41c-bdac-477f-94b8-2b41ba74100d") : secret "dns-default-metrics-tls" not found
Jul 28 23:22:19.602995 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.602895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e343f41c-bdac-477f-94b8-2b41ba74100d-tmp-dir\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.603115 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.603087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e343f41c-bdac-477f-94b8-2b41ba74100d-config-volume\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.606584 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.606557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4daeb74a-6c58-4c2b-836f-73d705ad0e5b-cert\") pod \"ingress-canary-kbzfw\" (UID: \"4daeb74a-6c58-4c2b-836f-73d705ad0e5b\") " pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.612324 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.612301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fsk\" (UniqueName: \"kubernetes.io/projected/e343f41c-bdac-477f-94b8-2b41ba74100d-kube-api-access-g4fsk\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:19.612589 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.612569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8q8\" (UniqueName: \"kubernetes.io/projected/4daeb74a-6c58-4c2b-836f-73d705ad0e5b-kube-api-access-cx8q8\") pod \"ingress-canary-kbzfw\" (UID: \"4daeb74a-6c58-4c2b-836f-73d705ad0e5b\") " pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.673681 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.673615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kbzfw"
Jul 28 23:22:19.803237 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.803087    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kbzfw"]
Jul 28 23:22:19.805290 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:19.805265    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4daeb74a_6c58_4c2b_836f_73d705ad0e5b.slice/crio-3b0a87553d437c602aed90af6961a04ce9bc32b7359e2647ed76c74add1694d3 WatchSource:0}: Error finding container 3b0a87553d437c602aed90af6961a04ce9bc32b7359e2647ed76c74add1694d3: Status 404 returned error can't find the container with id 3b0a87553d437c602aed90af6961a04ce9bc32b7359e2647ed76c74add1694d3
Jul 28 23:22:19.976117 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.976082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:19.979336 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.979313    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plrk7\""
Jul 28 23:22:19.979336 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:19.979324    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 23:22:20.104981 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.104898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:20.105116 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:20.105016    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 23:22:20.105116 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:20.105070    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls podName:e343f41c-bdac-477f-94b8-2b41ba74100d nodeName:}" failed. No retries permitted until 2026-07-28 23:22:21.105055162 +0000 UTC m=+36.607601671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls") pod "dns-default-5c9l5" (UID: "e343f41c-bdac-477f-94b8-2b41ba74100d") : secret "dns-default-metrics-tls" not found
Jul 28 23:22:20.240599 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.240565    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae37dc3-45bf-4c81-a4af-40683d4968ec" containerID="214beaa798e119bf6bcd121ad3ea7e060b768a5c2bb4202f329c61598343100e" exitCode=0
Jul 28 23:22:20.240764 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.240654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerDied","Data":"214beaa798e119bf6bcd121ad3ea7e060b768a5c2bb4202f329c61598343100e"}
Jul 28 23:22:20.241937 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.241699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kbzfw" event={"ID":"4daeb74a-6c58-4c2b-836f-73d705ad0e5b","Type":"ContainerStarted","Data":"3b0a87553d437c602aed90af6961a04ce9bc32b7359e2647ed76c74add1694d3"}
Jul 28 23:22:20.976347 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.976314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:20.976710 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.976547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:20.979172 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.979148    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 23:22:20.980197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.980162    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 23:22:20.980197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.980162    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 23:22:20.980553 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:20.980208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhxv7\""
Jul 28 23:22:21.111875 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:21.111836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:21.112026 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:21.112009    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 23:22:21.112110 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:21.112091    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls podName:e343f41c-bdac-477f-94b8-2b41ba74100d nodeName:}" failed. No retries permitted until 2026-07-28 23:22:23.112069274 +0000 UTC m=+38.614615785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls") pod "dns-default-5c9l5" (UID: "e343f41c-bdac-477f-94b8-2b41ba74100d") : secret "dns-default-metrics-tls" not found
Jul 28 23:22:21.248383 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:21.248301    2581 generic.go:356] "Generic (PLEG): container finished" podID="aae37dc3-45bf-4c81-a4af-40683d4968ec" containerID="531df6a0cf0b4506d86bb58c267c7eb9b42727d9627d7ed84e2f50701a7baa43" exitCode=0
Jul 28 23:22:21.248383 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:21.248347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerDied","Data":"531df6a0cf0b4506d86bb58c267c7eb9b42727d9627d7ed84e2f50701a7baa43"}
Jul 28 23:22:22.253649 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:22.253439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xxts" event={"ID":"aae37dc3-45bf-4c81-a4af-40683d4968ec","Type":"ContainerStarted","Data":"c9ab09dc364ab80dcefb32db32768c209029c1f786c32b98e49691adf7ecbca5"}
Jul 28 23:22:22.278777 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:22.277654    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8xxts" podStartSLOduration=8.469006207 podStartE2EDuration="37.277603312s" totalImagesPullingTime="28.808597105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:22.276482743 +0000 UTC m=+37.779029274" watchObservedRunningTime="2026-07-28 23:22:22.277603312 +0000 UTC m=+37.780149844"
Jul 28 23:22:23.127208 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:23.127180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:23.127309 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:23.127295    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 23:22:23.127365 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:23.127355    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls podName:e343f41c-bdac-477f-94b8-2b41ba74100d nodeName:}" failed. No retries permitted until 2026-07-28 23:22:27.127338266 +0000 UTC m=+42.629884774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls") pod "dns-default-5c9l5" (UID: "e343f41c-bdac-477f-94b8-2b41ba74100d") : secret "dns-default-metrics-tls" not found
Jul 28 23:22:23.258458 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:23.258418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kbzfw" event={"ID":"4daeb74a-6c58-4c2b-836f-73d705ad0e5b","Type":"ContainerStarted","Data":"161bbf0d2f71699bfb9704fbd3341db49bd8804521261e1c87f5499221accad1"}
Jul 28 23:22:23.274094 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:23.274050    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kbzfw" podStartSLOduration=1.781273132 podStartE2EDuration="4.27403399s" totalImagesPullingTime="2.492760858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:23.272979027 +0000 UTC m=+38.775525652" watchObservedRunningTime="2026-07-28 23:22:23.27403399 +0000 UTC m=+38.776580521"
Jul 28 23:22:27.157343 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:27.157299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:27.157844 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:27.157428    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 23:22:27.157844 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:27.157488    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls podName:e343f41c-bdac-477f-94b8-2b41ba74100d nodeName:}" failed. No retries permitted until 2026-07-28 23:22:35.157473842 +0000 UTC m=+50.660020351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls") pod "dns-default-5c9l5" (UID: "e343f41c-bdac-477f-94b8-2b41ba74100d") : secret "dns-default-metrics-tls" not found
Jul 28 23:22:34.330938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.330907    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"]
Jul 28 23:22:34.372698 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.372662    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"]
Jul 28 23:22:34.372893 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.372776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"
Jul 28 23:22:34.375483 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.375458    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 23:22:34.375619 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.375535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fc8jt\""
Jul 28 23:22:34.375619 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.375587    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 23:22:34.430606 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.430574    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"]
Jul 28 23:22:34.452089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.452050    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"]
Jul 28 23:22:34.452262 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.452175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"
Jul 28 23:22:34.454538 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.454516    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hng8m\""
Jul 28 23:22:34.462165 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.462145    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"]
Jul 28 23:22:34.462256 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.462172    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"]
Jul 28 23:22:34.462295 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.462252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.464508 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.464486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 23:22:34.464508 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.464502    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 23:22:34.464677 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.464523    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 23:22:34.464677 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.464533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dtgwn\""
Jul 28 23:22:34.464677 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.464528    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 23:22:34.507023 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.506989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4tzt\" (UniqueName: \"kubernetes.io/projected/0899f726-95b5-4000-8379-3753c736707d-kube-api-access-c4tzt\") pod \"volume-data-source-validator-549c6d54cf-685sw\" (UID: \"0899f726-95b5-4000-8379-3753c736707d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"
Jul 28 23:22:34.537274 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.537239    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"]
Jul 28 23:22:34.544193 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.544176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.547954 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.547916    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 23:22:34.547954 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.547925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fz42l\""
Jul 28 23:22:34.548250 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.548031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 23:22:34.548250 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.548161    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 23:22:34.548358 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.548325    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 23:22:34.550358 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.550338    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7869b868ff-lcdtr"]
Jul 28 23:22:34.558280 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.558260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"]
Jul 28 23:22:34.558527 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.558478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.564642 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.561498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gxdzj\""
Jul 28 23:22:34.564642 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.561724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 23:22:34.564642 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.562017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 23:22:34.564642 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.562213    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 23:22:34.575649 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.572014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7869b868ff-lcdtr"]
Jul 28 23:22:34.576774 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.576754    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 23:22:34.607824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.607801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmv7r\" (UniqueName: \"kubernetes.io/projected/71381373-e286-4cf0-8f0c-5048409119e9-kube-api-access-gmv7r\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.607938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.607843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71381373-e286-4cf0-8f0c-5048409119e9-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.607938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.607874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/4a70db46-b15f-4b00-be2e-b1ac64277a9d-kube-api-access-s8smc\") pod \"network-check-source-7d88cb55d8-th9gl\" (UID: \"4a70db46-b15f-4b00-be2e-b1ac64277a9d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"
Jul 28 23:22:34.607938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.607928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4tzt\" (UniqueName: \"kubernetes.io/projected/0899f726-95b5-4000-8379-3753c736707d-kube-api-access-c4tzt\") pod \"volume-data-source-validator-549c6d54cf-685sw\" (UID: \"0899f726-95b5-4000-8379-3753c736707d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"
Jul 28 23:22:34.608035 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.607943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71381373-e286-4cf0-8f0c-5048409119e9-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.618586 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.618564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4tzt\" (UniqueName: \"kubernetes.io/projected/0899f726-95b5-4000-8379-3753c736707d-kube-api-access-c4tzt\") pod \"volume-data-source-validator-549c6d54cf-685sw\" (UID: \"0899f726-95b5-4000-8379-3753c736707d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"
Jul 28 23:22:34.643381 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.643352    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"]
Jul 28 23:22:34.655787 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.655763    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"]
Jul 28 23:22:34.655915 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.655896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:34.658519 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.658332    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 23:22:34.658519 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.658429    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wxtzq\""
Jul 28 23:22:34.658519 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.658443    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 23:22:34.664110 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.664095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"]
Jul 28 23:22:34.664185 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.664121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"]
Jul 28 23:22:34.664221 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.664189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.666681 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.666661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mcbsc\""
Jul 28 23:22:34.666681 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.666673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 23:22:34.666831 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.666696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 23:22:34.667260 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.667240    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 23:22:34.667489 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.667466    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 23:22:34.681491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.681473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"
Jul 28 23:22:34.709165 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmv7r\" (UniqueName: \"kubernetes.io/projected/71381373-e286-4cf0-8f0c-5048409119e9-kube-api-access-gmv7r\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.709257 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c29d603-296e-47fc-8817-19def1f000c4-ca-trust-extracted\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.709257 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-bound-sa-token\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.709334 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktss\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-kube-api-access-hktss\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.709369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d35dd16-c51a-485c-acd1-88eaac50ac78-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.709409 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.709446 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71381373-e286-4cf0-8f0c-5048409119e9-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.709492 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.709527 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-installation-pull-secrets\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.709566 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-trusted-ca\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.709610 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/4a70db46-b15f-4b00-be2e-b1ac64277a9d-kube-api-access-s8smc\") pod \"network-check-source-7d88cb55d8-th9gl\" (UID: \"4a70db46-b15f-4b00-be2e-b1ac64277a9d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"
Jul 28 23:22:34.709683 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-registry-certificates\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.709683 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mx8d\" (UniqueName: \"kubernetes.io/projected/0d35dd16-c51a-485c-acd1-88eaac50ac78-kube-api-access-8mx8d\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.709779 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71381373-e286-4cf0-8f0c-5048409119e9-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.709779 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.709748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-image-registry-private-configuration\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.710217 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.710201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71381373-e286-4cf0-8f0c-5048409119e9-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.711683 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.711617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71381373-e286-4cf0-8f0c-5048409119e9-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.717159 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.717137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmv7r\" (UniqueName: \"kubernetes.io/projected/71381373-e286-4cf0-8f0c-5048409119e9-kube-api-access-gmv7r\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm\" (UID: \"71381373-e286-4cf0-8f0c-5048409119e9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.717483 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.717468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8smc\" (UniqueName: \"kubernetes.io/projected/4a70db46-b15f-4b00-be2e-b1ac64277a9d-kube-api-access-s8smc\") pod \"network-check-source-7d88cb55d8-th9gl\" (UID: \"4a70db46-b15f-4b00-be2e-b1ac64277a9d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"
Jul 28 23:22:34.760564 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.760532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"
Jul 28 23:22:34.770929 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.770901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"
Jul 28 23:22:34.810969 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.810881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mx8d\" (UniqueName: \"kubernetes.io/projected/0d35dd16-c51a-485c-acd1-88eaac50ac78-kube-api-access-8mx8d\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.810969 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.810962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-image-registry-private-configuration\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.810993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c29d603-296e-47fc-8817-19def1f000c4-ca-trust-extracted\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-bound-sa-token\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-config\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.811197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktss\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-kube-api-access-hktss\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-trusted-ca\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-installation-pull-secrets\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtmc\" (UniqueName: \"kubernetes.io/projected/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-kube-api-access-qqtmc\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-registry-certificates\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5af2f9d8-3754-4601-983c-f6216aac6937-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d35dd16-c51a-485c-acd1-88eaac50ac78-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.811496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.811476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.811889 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:34.811588    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 23:22:34.811889 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:34.811603    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7869b868ff-lcdtr: secret "image-registry-tls" not found
Jul 28 23:22:34.812676 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.812287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c29d603-296e-47fc-8817-19def1f000c4-ca-trust-extracted\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.812676 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.812587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-trusted-ca\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.812676 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:34.812591    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:34.812900 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.812692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw"]
Jul 28 23:22:34.812900 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:34.812716    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls podName:0d35dd16-c51a-485c-acd1-88eaac50ac78 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:35.312692628 +0000 UTC m=+50.815239152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tlmkg" (UID: "0d35dd16-c51a-485c-acd1-88eaac50ac78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:34.812900 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.812595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-registry-certificates\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.813386 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.813349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d35dd16-c51a-485c-acd1-88eaac50ac78-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.813498 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:34.813457    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls podName:1c29d603-296e-47fc-8817-19def1f000c4 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:35.313437499 +0000 UTC m=+50.815984016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls") pod "image-registry-7869b868ff-lcdtr" (UID: "1c29d603-296e-47fc-8817-19def1f000c4") : secret "image-registry-tls" not found
Jul 28 23:22:34.814302 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.814281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-installation-pull-secrets\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.815135 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.815114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-image-registry-private-configuration\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.821803 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.821762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mx8d\" (UniqueName: \"kubernetes.io/projected/0d35dd16-c51a-485c-acd1-88eaac50ac78-kube-api-access-8mx8d\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:34.821803 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.821777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-bound-sa-token\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.821949 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.821927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktss\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-kube-api-access-hktss\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:34.912287 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.912253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-config\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.912287 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.912302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:34.912548 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.912322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.912548 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.912344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtmc\" (UniqueName: \"kubernetes.io/projected/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-kube-api-access-qqtmc\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.912548 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.912373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5af2f9d8-3754-4601-983c-f6216aac6937-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:34.912786 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:34.912757    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 23:22:34.912916 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:34.912851    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert podName:5af2f9d8-3754-4601-983c-f6216aac6937 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:35.412830211 +0000 UTC m=+50.915376735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-h4vwh" (UID: "5af2f9d8-3754-4601-983c-f6216aac6937") : secret "networking-console-plugin-cert" not found
Jul 28 23:22:34.913283 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.912988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-config\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.913283 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.913085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5af2f9d8-3754-4601-983c-f6216aac6937-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:34.915487 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.915457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-serving-cert\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.917965 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.917946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl"]
Jul 28 23:22:34.921370 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:34.921344    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a70db46_b15f_4b00_be2e_b1ac64277a9d.slice/crio-08319f1e8751d0b4e6382069ec9731df398143da9cb8a1b2aeaf2ea16ae92543 WatchSource:0}: Error finding container 08319f1e8751d0b4e6382069ec9731df398143da9cb8a1b2aeaf2ea16ae92543: Status 404 returned error can't find the container with id 08319f1e8751d0b4e6382069ec9731df398143da9cb8a1b2aeaf2ea16ae92543
Jul 28 23:22:34.922845 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.922823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtmc\" (UniqueName: \"kubernetes.io/projected/619deb1c-42ab-4f7f-b489-ff4e04e0fd00-kube-api-access-qqtmc\") pod \"service-ca-operator-5d6c75f84f-ltnqh\" (UID: \"619deb1c-42ab-4f7f-b489-ff4e04e0fd00\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:34.941851 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.941814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm"]
Jul 28 23:22:34.945164 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:34.945138    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71381373_e286_4cf0_8f0c_5048409119e9.slice/crio-53a4157e7a8a9a09d4bdfca7e761315dd41e2efd227156a288721ac0f7f56fb6 WatchSource:0}: Error finding container 53a4157e7a8a9a09d4bdfca7e761315dd41e2efd227156a288721ac0f7f56fb6: Status 404 returned error can't find the container with id 53a4157e7a8a9a09d4bdfca7e761315dd41e2efd227156a288721ac0f7f56fb6
Jul 28 23:22:34.975564 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:34.975538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"
Jul 28 23:22:35.123966 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.123856    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh"]
Jul 28 23:22:35.137872 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:35.137846    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619deb1c_42ab_4f7f_b489_ff4e04e0fd00.slice/crio-bc431ff5df077fa66a2f22735a87bda7eb6276fb92269432003f99dcb9b1005e WatchSource:0}: Error finding container bc431ff5df077fa66a2f22735a87bda7eb6276fb92269432003f99dcb9b1005e: Status 404 returned error can't find the container with id bc431ff5df077fa66a2f22735a87bda7eb6276fb92269432003f99dcb9b1005e
Jul 28 23:22:35.214708 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.214681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:35.214816 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.214788    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 23:22:35.214861 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.214849    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls podName:e343f41c-bdac-477f-94b8-2b41ba74100d nodeName:}" failed. No retries permitted until 2026-07-28 23:22:51.214834013 +0000 UTC m=+66.717380522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls") pod "dns-default-5c9l5" (UID: "e343f41c-bdac-477f-94b8-2b41ba74100d") : secret "dns-default-metrics-tls" not found
Jul 28 23:22:35.286530 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.286497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh" event={"ID":"619deb1c-42ab-4f7f-b489-ff4e04e0fd00","Type":"ContainerStarted","Data":"bc431ff5df077fa66a2f22735a87bda7eb6276fb92269432003f99dcb9b1005e"}
Jul 28 23:22:35.287614 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.287593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm" event={"ID":"71381373-e286-4cf0-8f0c-5048409119e9","Type":"ContainerStarted","Data":"53a4157e7a8a9a09d4bdfca7e761315dd41e2efd227156a288721ac0f7f56fb6"}
Jul 28 23:22:35.288652 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.288621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl" event={"ID":"4a70db46-b15f-4b00-be2e-b1ac64277a9d","Type":"ContainerStarted","Data":"08319f1e8751d0b4e6382069ec9731df398143da9cb8a1b2aeaf2ea16ae92543"}
Jul 28 23:22:35.289591 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.289571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw" event={"ID":"0899f726-95b5-4000-8379-3753c736707d","Type":"ContainerStarted","Data":"0da41614944408bb17d3815b4d4bee2418edb045ba6e060be86a93240d6617ab"}
Jul 28 23:22:35.315139 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.315115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:35.315257 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.315151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:35.315257 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.315236    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 23:22:35.315257 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.315246    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7869b868ff-lcdtr: secret "image-registry-tls" not found
Jul 28 23:22:35.315353 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.315253    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:35.315353 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.315288    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls podName:1c29d603-296e-47fc-8817-19def1f000c4 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:36.315276339 +0000 UTC m=+51.817822848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls") pod "image-registry-7869b868ff-lcdtr" (UID: "1c29d603-296e-47fc-8817-19def1f000c4") : secret "image-registry-tls" not found
Jul 28 23:22:35.315353 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.315306    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls podName:0d35dd16-c51a-485c-acd1-88eaac50ac78 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:36.315292717 +0000 UTC m=+51.817839227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tlmkg" (UID: "0d35dd16-c51a-485c-acd1-88eaac50ac78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:35.416180 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:35.416117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:35.416470 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.416214    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 23:22:35.416470 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:35.416260    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert podName:5af2f9d8-3754-4601-983c-f6216aac6937 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:36.416248354 +0000 UTC m=+51.918794864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-h4vwh" (UID: "5af2f9d8-3754-4601-983c-f6216aac6937") : secret "networking-console-plugin-cert" not found
Jul 28 23:22:36.324539 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:36.324496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:36.324808 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:36.324565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:36.324889 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:36.324816    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:36.324889 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:36.324875    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 23:22:36.324988 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:36.324892    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls podName:0d35dd16-c51a-485c-acd1-88eaac50ac78 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:38.324870143 +0000 UTC m=+53.827416666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tlmkg" (UID: "0d35dd16-c51a-485c-acd1-88eaac50ac78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:36.324988 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:36.324892    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7869b868ff-lcdtr: secret "image-registry-tls" not found
Jul 28 23:22:36.324988 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:36.324935    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls podName:1c29d603-296e-47fc-8817-19def1f000c4 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:38.324922547 +0000 UTC m=+53.827469079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls") pod "image-registry-7869b868ff-lcdtr" (UID: "1c29d603-296e-47fc-8817-19def1f000c4") : secret "image-registry-tls" not found
Jul 28 23:22:36.425574 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:36.425432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:36.426046 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:36.425657    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 23:22:36.426046 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:36.425726    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert podName:5af2f9d8-3754-4601-983c-f6216aac6937 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:38.425706599 +0000 UTC m=+53.928253108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-h4vwh" (UID: "5af2f9d8-3754-4601-983c-f6216aac6937") : secret "networking-console-plugin-cert" not found
Jul 28 23:22:37.297981 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:37.297940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw" event={"ID":"0899f726-95b5-4000-8379-3753c736707d","Type":"ContainerStarted","Data":"bfa25ca4af4336186acc3fc5000a7affb2fbe0024662555ae02f113dab270b36"}
Jul 28 23:22:37.311998 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:37.311942    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-685sw" podStartSLOduration=1.495052517 podStartE2EDuration="3.311923076s" totalImagesPullingTime="1.816870559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:37.310970937 +0000 UTC m=+52.813517470" watchObservedRunningTime="2026-07-28 23:22:37.311923076 +0000 UTC m=+52.814469608"
Jul 28 23:22:38.280008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.279976    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jq58v"]
Jul 28 23:22:38.283458 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.283438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.286069 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.286015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 23:22:38.286283 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.286259    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 23:22:38.286362 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.286270    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xh4g8\""
Jul 28 23:22:38.287286 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.287267    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 23:22:38.287388 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.287304    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 23:22:38.295535 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.295512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jq58v"]
Jul 28 23:22:38.342326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.342289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:38.342504 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.342347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:38.342504 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.342497    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 23:22:38.342600 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.342514    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7869b868ff-lcdtr: secret "image-registry-tls" not found
Jul 28 23:22:38.342600 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.342568    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls podName:1c29d603-296e-47fc-8817-19def1f000c4 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:42.342551465 +0000 UTC m=+57.845097974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls") pod "image-registry-7869b868ff-lcdtr" (UID: "1c29d603-296e-47fc-8817-19def1f000c4") : secret "image-registry-tls" not found
Jul 28 23:22:38.342948 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.342854    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:38.342948 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.342924    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls podName:0d35dd16-c51a-485c-acd1-88eaac50ac78 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:42.342907071 +0000 UTC m=+57.845453595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tlmkg" (UID: "0d35dd16-c51a-485c-acd1-88eaac50ac78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:38.443529 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.443491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.443746 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.443567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24h4l\" (UniqueName: \"kubernetes.io/projected/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-api-access-24h4l\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.443746 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.443612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-crio-socket\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.443746 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.443703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:38.443918 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.443780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.443918 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.443822    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 23:22:38.443918 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.443887    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert podName:5af2f9d8-3754-4601-983c-f6216aac6937 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:42.443864492 +0000 UTC m=+57.946411006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-h4vwh" (UID: "5af2f9d8-3754-4601-983c-f6216aac6937") : secret "networking-console-plugin-cert" not found
Jul 28 23:22:38.444079 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.443941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-data-volume\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.544682 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.544584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-data-volume\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.544845 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.544701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.544845 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.544748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24h4l\" (UniqueName: \"kubernetes.io/projected/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-api-access-24h4l\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.544845 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.544780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-crio-socket\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.544845 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.544822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.545042 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.544844    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 23:22:38.545042 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.544930    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:39.044907547 +0000 UTC m=+54.547454069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : configmap "kube-rbac-proxy" not found
Jul 28 23:22:38.545042 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.544937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-crio-socket\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.545042 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.544953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-data-volume\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:38.545042 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.544942    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:38.545042 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:38.545030    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:39.045013281 +0000 UTC m=+54.547559790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:38.552853 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:38.552828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24h4l\" (UniqueName: \"kubernetes.io/projected/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-api-access-24h4l\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:39.048874 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.048833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:39.049074 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.048977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:39.049074 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:39.048982    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:39.049193 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:39.049021    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 23:22:39.049193 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:39.049113    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:40.049093849 +0000 UTC m=+55.551640363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:39.049193 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:39.049138    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:40.049124874 +0000 UTC m=+55.551671399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : configmap "kube-rbac-proxy" not found
Jul 28 23:22:39.304902 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.304790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm" event={"ID":"71381373-e286-4cf0-8f0c-5048409119e9","Type":"ContainerStarted","Data":"414c58bb6b38a72cbe22b79dd66312b38107ef52ff61a7530054250691d94b36"}
Jul 28 23:22:39.306828 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.306796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl" event={"ID":"4a70db46-b15f-4b00-be2e-b1ac64277a9d","Type":"ContainerStarted","Data":"0c35bb26c30bf17be5893393ddbc286be0be4f582a233e28a932d01128e78f8c"}
Jul 28 23:22:39.308422 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.308398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh" event={"ID":"619deb1c-42ab-4f7f-b489-ff4e04e0fd00","Type":"ContainerStarted","Data":"c60c65ff9da68dfea076065f112005c805020f3a1e0088b5412709afc3ff6070"}
Jul 28 23:22:39.319992 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.319947    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm" podStartSLOduration=1.315674512 podStartE2EDuration="5.319934283s" totalImagesPullingTime="4.004259771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:39.319060702 +0000 UTC m=+54.821607232" watchObservedRunningTime="2026-07-28 23:22:39.319934283 +0000 UTC m=+54.822480792"
Jul 28 23:22:39.373063 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.372956    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-th9gl" podStartSLOduration=1.33218719 podStartE2EDuration="5.372934287s" totalImagesPullingTime="4.040747097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:39.372538665 +0000 UTC m=+54.875085197" watchObservedRunningTime="2026-07-28 23:22:39.372934287 +0000 UTC m=+54.875480819"
Jul 28 23:22:39.373262 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:39.373212    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh" podStartSLOduration=1.55780947 podStartE2EDuration="5.373204152s" totalImagesPullingTime="3.815394682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:39.338803138 +0000 UTC m=+54.841349667" watchObservedRunningTime="2026-07-28 23:22:39.373204152 +0000 UTC m=+54.875750684"
Jul 28 23:22:40.057533 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:40.057496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:40.057743 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:40.057567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:40.057743 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:40.057656    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 23:22:40.057743 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:40.057729    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:42.057711799 +0000 UTC m=+57.560258309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : configmap "kube-rbac-proxy" not found
Jul 28 23:22:40.057743 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:40.057733    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:40.057905 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:40.057779    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:42.057766593 +0000 UTC m=+57.560313101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:40.062065 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:40.062048    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w5rtn_9125b4bf-4b57-4adb-a1c6-9eb94d4525db/dns-node-resolver/0.log"
Jul 28 23:22:41.461696 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:41.461664    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvngb_0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91/node-ca/0.log"
Jul 28 23:22:42.061673 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:42.061644    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kbzfw_4daeb74a-6c58-4c2b-836f-73d705ad0e5b/serve-healthcheck-canary/0.log"
Jul 28 23:22:42.071861 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:42.071826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:42.071996 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:42.071924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:42.071996 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.071940    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 23:22:42.071996 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.071997    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:46.071973909 +0000 UTC m=+61.574520417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : configmap "kube-rbac-proxy" not found
Jul 28 23:22:42.072144 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.072044    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:42.072144 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.072103    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:46.072087667 +0000 UTC m=+61.574634193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:42.374017 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:42.373980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:42.374174 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:42.374035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:42.374174 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.374154    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:42.374242 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.374175    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 23:22:42.374242 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.374195    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7869b868ff-lcdtr: secret "image-registry-tls" not found
Jul 28 23:22:42.374242 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.374215    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls podName:0d35dd16-c51a-485c-acd1-88eaac50ac78 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:50.374197676 +0000 UTC m=+65.876744384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tlmkg" (UID: "0d35dd16-c51a-485c-acd1-88eaac50ac78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:42.374242 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.374237    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls podName:1c29d603-296e-47fc-8817-19def1f000c4 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:50.374223227 +0000 UTC m=+65.876769736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls") pod "image-registry-7869b868ff-lcdtr" (UID: "1c29d603-296e-47fc-8817-19def1f000c4") : secret "image-registry-tls" not found
Jul 28 23:22:42.474730 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:42.474690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:42.475111 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.474831    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 23:22:42.475111 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:42.474891    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert podName:5af2f9d8-3754-4601-983c-f6216aac6937 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:50.474877327 +0000 UTC m=+65.977423836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-h4vwh" (UID: "5af2f9d8-3754-4601-983c-f6216aac6937") : secret "networking-console-plugin-cert" not found
Jul 28 23:22:45.233323 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:45.233298    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6kt"
Jul 28 23:22:46.104042 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:46.104008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:46.104218 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:46.104137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:46.104218 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:46.104161    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:46.104218 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:46.104210    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:54.104196356 +0000 UTC m=+69.606742866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:46.104349 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:46.104248    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 23:22:46.104349 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:46.104295    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:22:54.10427958 +0000 UTC m=+69.606826102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : configmap "kube-rbac-proxy" not found
Jul 28 23:22:50.441861 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.441822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:50.442254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.441967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:22:50.442254 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:50.442059    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:50.442254 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:50.442114    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls podName:0d35dd16-c51a-485c-acd1-88eaac50ac78 nodeName:}" failed. No retries permitted until 2026-07-28 23:23:06.442100062 +0000 UTC m=+81.944646576 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tlmkg" (UID: "0d35dd16-c51a-485c-acd1-88eaac50ac78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 23:22:50.444351 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.444316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"image-registry-7869b868ff-lcdtr\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") " pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:50.483903 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.483872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gxdzj\""
Jul 28 23:22:50.491910 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.491885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:50.543392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.543353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:22:50.543550 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:50.543527    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 23:22:50.543649 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:50.543606    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert podName:5af2f9d8-3754-4601-983c-f6216aac6937 nodeName:}" failed. No retries permitted until 2026-07-28 23:23:06.543586562 +0000 UTC m=+82.046133080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-h4vwh" (UID: "5af2f9d8-3754-4601-983c-f6216aac6937") : secret "networking-console-plugin-cert" not found
Jul 28 23:22:50.618753 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.618544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7869b868ff-lcdtr"]
Jul 28 23:22:50.620598 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:50.620567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c29d603_296e_47fc_8817_19def1f000c4.slice/crio-e0d9b38156809f2c0c227f63991947d17bc2a2419ee517f4f6cb9faf57ef14fc WatchSource:0}: Error finding container e0d9b38156809f2c0c227f63991947d17bc2a2419ee517f4f6cb9faf57ef14fc: Status 404 returned error can't find the container with id e0d9b38156809f2c0c227f63991947d17bc2a2419ee517f4f6cb9faf57ef14fc
Jul 28 23:22:50.746131 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.746040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:22:50.746131 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.746123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:50.746310 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.746258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:50.748589 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.748556    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 23:22:50.748702 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.748618    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 23:22:50.748702 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.748653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwvd\" (UniqueName: \"kubernetes.io/projected/5e6fa954-a3ac-48aa-97de-5d2b150c427a-kube-api-access-thwvd\") pod \"network-check-target-cbw8q\" (UID: \"5e6fa954-a3ac-48aa-97de-5d2b150c427a\") " pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:50.756619 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:50.756601    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 23:22:50.756688 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:50.756667    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs podName:0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6 nodeName:}" failed. No retries permitted until 2026-07-28 23:23:54.756652959 +0000 UTC m=+130.259199468 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs") pod "network-metrics-daemon-m968q" (UID: "0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6") : secret "metrics-daemon-secret" not found
Jul 28 23:22:50.758584 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.758564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17ed82b0-64ac-4fe9-aff4-3ef02f1476a0-original-pull-secret\") pod \"global-pull-secret-syncer-fkq46\" (UID: \"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0\") " pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:50.991620 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.991590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhxv7\""
Jul 28 23:22:50.994616 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.994602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkq46"
Jul 28 23:22:50.998724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:50.998662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:51.138540 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.138510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fkq46"]
Jul 28 23:22:51.141831 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:51.141770    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ed82b0_64ac_4fe9_aff4_3ef02f1476a0.slice/crio-14ed6aa451b5e74e2d54194365bf0ab653fee60ee9d39181529d2ee05b6c94de WatchSource:0}: Error finding container 14ed6aa451b5e74e2d54194365bf0ab653fee60ee9d39181529d2ee05b6c94de: Status 404 returned error can't find the container with id 14ed6aa451b5e74e2d54194365bf0ab653fee60ee9d39181529d2ee05b6c94de
Jul 28 23:22:51.169591 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.169566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cbw8q"]
Jul 28 23:22:51.172909 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:51.172886    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6fa954_a3ac_48aa_97de_5d2b150c427a.slice/crio-0f35d1a27ca33074f131f98397a562b13020b71b0a86c6a6fb6ed5cfebe204fa WatchSource:0}: Error finding container 0f35d1a27ca33074f131f98397a562b13020b71b0a86c6a6fb6ed5cfebe204fa: Status 404 returned error can't find the container with id 0f35d1a27ca33074f131f98397a562b13020b71b0a86c6a6fb6ed5cfebe204fa
Jul 28 23:22:51.249338 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.249280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:51.251280 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.251262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e343f41c-bdac-477f-94b8-2b41ba74100d-metrics-tls\") pod \"dns-default-5c9l5\" (UID: \"e343f41c-bdac-477f-94b8-2b41ba74100d\") " pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:51.338784 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.338750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fkq46" event={"ID":"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0","Type":"ContainerStarted","Data":"14ed6aa451b5e74e2d54194365bf0ab653fee60ee9d39181529d2ee05b6c94de"}
Jul 28 23:22:51.340080 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.340052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbw8q" event={"ID":"5e6fa954-a3ac-48aa-97de-5d2b150c427a","Type":"ContainerStarted","Data":"04e7ecd46c8126a7fe7e648bbf385b5008ad7051a20af053387d26bf0c7262e9"}
Jul 28 23:22:51.340261 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.340088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cbw8q" event={"ID":"5e6fa954-a3ac-48aa-97de-5d2b150c427a","Type":"ContainerStarted","Data":"0f35d1a27ca33074f131f98397a562b13020b71b0a86c6a6fb6ed5cfebe204fa"}
Jul 28 23:22:51.340261 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.340164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:22:51.341563 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.341541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr" event={"ID":"1c29d603-296e-47fc-8817-19def1f000c4","Type":"ContainerStarted","Data":"e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4"}
Jul 28 23:22:51.341697 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.341571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr" event={"ID":"1c29d603-296e-47fc-8817-19def1f000c4","Type":"ContainerStarted","Data":"e0d9b38156809f2c0c227f63991947d17bc2a2419ee517f4f6cb9faf57ef14fc"}
Jul 28 23:22:51.341697 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.341663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:22:51.357408 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.354614    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cbw8q" podStartSLOduration=66.354597442 podStartE2EDuration="1m6.354597442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:51.353142056 +0000 UTC m=+66.855688588" watchObservedRunningTime="2026-07-28 23:22:51.354597442 +0000 UTC m=+66.857143974"
Jul 28 23:22:51.371602 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.371561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr" podStartSLOduration=17.371549717 podStartE2EDuration="17.371549717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:51.370935796 +0000 UTC m=+66.873482356" watchObservedRunningTime="2026-07-28 23:22:51.371549717 +0000 UTC m=+66.874096248"
Jul 28 23:22:51.471246 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.471221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7kmkd\""
Jul 28 23:22:51.479218 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.479202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:51.618876 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:51.618847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c9l5"]
Jul 28 23:22:51.622011 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:22:51.621982    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode343f41c_bdac_477f_94b8_2b41ba74100d.slice/crio-afc5b7e024d60a99cefe7db97cf50634ae75ec1afcf8222388647b28a1bc9c24 WatchSource:0}: Error finding container afc5b7e024d60a99cefe7db97cf50634ae75ec1afcf8222388647b28a1bc9c24: Status 404 returned error can't find the container with id afc5b7e024d60a99cefe7db97cf50634ae75ec1afcf8222388647b28a1bc9c24
Jul 28 23:22:52.347736 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:52.347682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c9l5" event={"ID":"e343f41c-bdac-477f-94b8-2b41ba74100d","Type":"ContainerStarted","Data":"afc5b7e024d60a99cefe7db97cf50634ae75ec1afcf8222388647b28a1bc9c24"}
Jul 28 23:22:54.176113 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:54.176078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:54.176560 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:54.176148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:54.176560 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:54.176230    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:54.176560 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:22:54.176298    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls podName:39d5eecd-36b6-4c80-94bf-47cb2888e4f8 nodeName:}" failed. No retries permitted until 2026-07-28 23:23:10.176281702 +0000 UTC m=+85.678828212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jq58v" (UID: "39d5eecd-36b6-4c80-94bf-47cb2888e4f8") : secret "insights-runtime-extractor-tls" not found
Jul 28 23:22:54.176710 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:54.176587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:22:54.356825 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:54.356788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c9l5" event={"ID":"e343f41c-bdac-477f-94b8-2b41ba74100d","Type":"ContainerStarted","Data":"3d63caf6f0847f9344cd4613920fc14d0a1b98a23b191fc5a7f05b52b4deb6a4"}
Jul 28 23:22:54.357013 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:54.356853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c9l5" event={"ID":"e343f41c-bdac-477f-94b8-2b41ba74100d","Type":"ContainerStarted","Data":"d3a2f7e4a8281b7d8cc52e12e2d44d64254993a6abf34f57c0cd7f6072e5d9a3"}
Jul 28 23:22:54.357013 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:54.356978    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:22:54.373211 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:54.373159    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5c9l5" podStartSLOduration=33.179136366 podStartE2EDuration="35.373139484s" totalImagesPullingTime="2.194003118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:54.372599864 +0000 UTC m=+69.875146396" watchObservedRunningTime="2026-07-28 23:22:54.373139484 +0000 UTC m=+69.875686014"
Jul 28 23:22:57.368906 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:57.368867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fkq46" event={"ID":"17ed82b0-64ac-4fe9-aff4-3ef02f1476a0","Type":"ContainerStarted","Data":"a3bd435bf2f903a722f27a87c2c9884b96c8c72f5245c350328e53c75d870e00"}
Jul 28 23:22:57.382867 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:22:57.382824    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fkq46" podStartSLOduration=66.990161943 podStartE2EDuration="1m12.382809379s" totalImagesPullingTime="5.392647436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:22:57.382102611 +0000 UTC m=+72.884649143" watchObservedRunningTime="2026-07-28 23:22:57.382809379 +0000 UTC m=+72.885355910"
Jul 28 23:23:01.582828 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.582793    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"]
Jul 28 23:23:01.585900 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.585883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.589259 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.589239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 23:23:01.589352 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.589292    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 23:23:01.589352 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.589324    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 23:23:01.589438 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.589375    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 23:23:01.596129 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.596109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"]
Jul 28 23:23:01.634931 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.634899    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7869b868ff-lcdtr"]
Jul 28 23:23:01.641863 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.641829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ac3b674-9b48-4c38-9289-39a44155d4fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.642015 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.641904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxqx\" (UniqueName: \"kubernetes.io/projected/8ac3b674-9b48-4c38-9289-39a44155d4fc-kube-api-access-vpxqx\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.642015 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.641962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ac3b674-9b48-4c38-9289-39a44155d4fc-tmp\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.693043 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.693013    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8b678dc9-6rcqc"]
Jul 28 23:23:01.696133 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.696109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.719991 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.719961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8b678dc9-6rcqc"]
Jul 28 23:23:01.742469 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxqx\" (UniqueName: \"kubernetes.io/projected/8ac3b674-9b48-4c38-9289-39a44155d4fc-kube-api-access-vpxqx\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.742617 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-bound-sa-token\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.742617 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ac3b674-9b48-4c38-9289-39a44155d4fc-tmp\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.742707 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-registry-tls\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.742707 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-installation-pull-secrets\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.742781 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srl82\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-kube-api-access-srl82\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.742819 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ac3b674-9b48-4c38-9289-39a44155d4fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.742857 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-ca-trust-extracted\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.742895 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-registry-certificates\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.742927 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-image-registry-private-configuration\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.742970 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.742945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-trusted-ca\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.743034 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.743019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ac3b674-9b48-4c38-9289-39a44155d4fc-tmp\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.745210 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.745177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ac3b674-9b48-4c38-9289-39a44155d4fc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.750463 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.750437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxqx\" (UniqueName: \"kubernetes.io/projected/8ac3b674-9b48-4c38-9289-39a44155d4fc-kube-api-access-vpxqx\") pod \"klusterlet-addon-workmgr-7465fddd5c-8jkxc\" (UID: \"8ac3b674-9b48-4c38-9289-39a44155d4fc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:01.843870 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.843796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-bound-sa-token\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.843870 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.843840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-registry-tls\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.844077 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.843879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-installation-pull-secrets\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.844077 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.843898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srl82\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-kube-api-access-srl82\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.844077 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.843949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-ca-trust-extracted\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.844077 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.843973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-registry-certificates\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.844286 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.844150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-image-registry-private-configuration\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.844286 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.844179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-trusted-ca\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.844516 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.844493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-ca-trust-extracted\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.845464 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.845439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-registry-certificates\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.845915 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.845894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-trusted-ca\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.846223 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.846196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-registry-tls\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.846371 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.846355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-image-registry-private-configuration\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.846488 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.846469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-installation-pull-secrets\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.853850 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.853821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-bound-sa-token\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.853938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.853834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srl82\" (UniqueName: \"kubernetes.io/projected/19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf-kube-api-access-srl82\") pod \"image-registry-5d8b678dc9-6rcqc\" (UID: \"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf\") " pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:01.895183 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:01.895158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:02.004595 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.004555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:02.016424 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.016403    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"]
Jul 28 23:23:02.018838 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:02.018809    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac3b674_9b48_4c38_9289_39a44155d4fc.slice/crio-3482468d17b8f640b1cc14e78af105d590ea063c55225ff21d932991f39d055c WatchSource:0}: Error finding container 3482468d17b8f640b1cc14e78af105d590ea063c55225ff21d932991f39d055c: Status 404 returned error can't find the container with id 3482468d17b8f640b1cc14e78af105d590ea063c55225ff21d932991f39d055c
Jul 28 23:23:02.129302 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.129272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8b678dc9-6rcqc"]
Jul 28 23:23:02.133679 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:02.133650    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f3ce17_f120_42c3_9a4f_bf4aa94c7cbf.slice/crio-7768c2a8b4eeef29e640169321d9717bad7e12800444557aa8de16d8b75dbd6e WatchSource:0}: Error finding container 7768c2a8b4eeef29e640169321d9717bad7e12800444557aa8de16d8b75dbd6e: Status 404 returned error can't find the container with id 7768c2a8b4eeef29e640169321d9717bad7e12800444557aa8de16d8b75dbd6e
Jul 28 23:23:02.386505 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.386405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc" event={"ID":"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf","Type":"ContainerStarted","Data":"2bf2e084405c3a33af381e8bf3d3969b9425dc6278b0d29670704990e111f7fb"}
Jul 28 23:23:02.386505 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.386467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc" event={"ID":"19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf","Type":"ContainerStarted","Data":"7768c2a8b4eeef29e640169321d9717bad7e12800444557aa8de16d8b75dbd6e"}
Jul 28 23:23:02.386505 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.386494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:02.387583 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.387558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc" event={"ID":"8ac3b674-9b48-4c38-9289-39a44155d4fc","Type":"ContainerStarted","Data":"3482468d17b8f640b1cc14e78af105d590ea063c55225ff21d932991f39d055c"}
Jul 28 23:23:02.404595 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:02.404540    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc" podStartSLOduration=1.404525823 podStartE2EDuration="1.404525823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:02.403596972 +0000 UTC m=+77.906143506" watchObservedRunningTime="2026-07-28 23:23:02.404525823 +0000 UTC m=+77.907072354"
Jul 28 23:23:04.363464 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:04.363435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5c9l5"
Jul 28 23:23:06.406261 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.406224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc" event={"ID":"8ac3b674-9b48-4c38-9289-39a44155d4fc","Type":"ContainerStarted","Data":"0486d8af8650314c8d6d6bdd4a8dc20089431d2cd52526af8221b497e67dbd85"}
Jul 28 23:23:06.406727 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.406436    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:06.408074 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.408056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc"
Jul 28 23:23:06.437303 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.437247    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7465fddd5c-8jkxc" podStartSLOduration=1.642736891 podStartE2EDuration="5.437231384s" totalImagesPullingTime="3.794494493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:06.422577444 +0000 UTC m=+81.925123975" watchObservedRunningTime="2026-07-28 23:23:06.437231384 +0000 UTC m=+81.939777915"
Jul 28 23:23:06.491379 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.491344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:23:06.493827 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.493799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d35dd16-c51a-485c-acd1-88eaac50ac78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tlmkg\" (UID: \"0d35dd16-c51a-485c-acd1-88eaac50ac78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:23:06.592500 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.592459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:23:06.594943 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.594916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5af2f9d8-3754-4601-983c-f6216aac6937-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-h4vwh\" (UID: \"5af2f9d8-3754-4601-983c-f6216aac6937\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:23:06.658741 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.658668    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fz42l\""
Jul 28 23:23:06.666691 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.666672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"
Jul 28 23:23:06.768672 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.768645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wxtzq\""
Jul 28 23:23:06.775887 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.775860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"
Jul 28 23:23:06.794562 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.794538    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg"]
Jul 28 23:23:06.797246 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:06.797218    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d35dd16_c51a_485c_acd1_88eaac50ac78.slice/crio-31ee2e0f39f54001d2d4612ff010e339ebb32e4da4d609791f76e57ea3a60e14 WatchSource:0}: Error finding container 31ee2e0f39f54001d2d4612ff010e339ebb32e4da4d609791f76e57ea3a60e14: Status 404 returned error can't find the container with id 31ee2e0f39f54001d2d4612ff010e339ebb32e4da4d609791f76e57ea3a60e14
Jul 28 23:23:06.898752 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:06.898723    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh"]
Jul 28 23:23:06.901798 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:06.901773    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af2f9d8_3754_4601_983c_f6216aac6937.slice/crio-deddc1213b7a92e506b6e5e94d1a7f07bb7bdd9b6282bb3a02af6117c48022ac WatchSource:0}: Error finding container deddc1213b7a92e506b6e5e94d1a7f07bb7bdd9b6282bb3a02af6117c48022ac: Status 404 returned error can't find the container with id deddc1213b7a92e506b6e5e94d1a7f07bb7bdd9b6282bb3a02af6117c48022ac
Jul 28 23:23:07.411722 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:07.411585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh" event={"ID":"5af2f9d8-3754-4601-983c-f6216aac6937","Type":"ContainerStarted","Data":"deddc1213b7a92e506b6e5e94d1a7f07bb7bdd9b6282bb3a02af6117c48022ac"}
Jul 28 23:23:07.412860 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:07.412816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg" event={"ID":"0d35dd16-c51a-485c-acd1-88eaac50ac78","Type":"ContainerStarted","Data":"31ee2e0f39f54001d2d4612ff010e339ebb32e4da4d609791f76e57ea3a60e14"}
Jul 28 23:23:09.421481 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:09.421437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh" event={"ID":"5af2f9d8-3754-4601-983c-f6216aac6937","Type":"ContainerStarted","Data":"701a3b3bd28ae21f9d920952a66a850fa331f291b3429e2fbd482d1ec3bef3b8"}
Jul 28 23:23:09.422951 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:09.422927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg" event={"ID":"0d35dd16-c51a-485c-acd1-88eaac50ac78","Type":"ContainerStarted","Data":"19ef8aab90fba1d8d0ec8cf26da8567e344c4f9d035b72f9396587016a97198b"}
Jul 28 23:23:09.435777 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:09.435737    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-h4vwh" podStartSLOduration=33.549864418 podStartE2EDuration="35.435722502s" totalImagesPullingTime="1.885858084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:09.435246423 +0000 UTC m=+84.937792954" watchObservedRunningTime="2026-07-28 23:23:09.435722502 +0000 UTC m=+84.938269010"
Jul 28 23:23:09.449314 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:09.449271    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tlmkg" podStartSLOduration=33.454915494 podStartE2EDuration="35.44925343s" totalImagesPullingTime="1.994337936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:09.448441546 +0000 UTC m=+84.950988077" watchObservedRunningTime="2026-07-28 23:23:09.44925343 +0000 UTC m=+84.951799965"
Jul 28 23:23:10.225871 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:10.225823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:23:10.228044 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:10.228014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39d5eecd-36b6-4c80-94bf-47cb2888e4f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jq58v\" (UID: \"39d5eecd-36b6-4c80-94bf-47cb2888e4f8\") " pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:23:10.395872 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:10.395839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xh4g8\""
Jul 28 23:23:10.403884 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:10.403864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jq58v"
Jul 28 23:23:10.560263 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:10.560228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jq58v"]
Jul 28 23:23:10.562743 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:10.562713    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d5eecd_36b6_4c80_94bf_47cb2888e4f8.slice/crio-3fa679615d34813d40c5a7dc5829ccc4a02b4a764a848af569e9d52e7ff276a9 WatchSource:0}: Error finding container 3fa679615d34813d40c5a7dc5829ccc4a02b4a764a848af569e9d52e7ff276a9: Status 404 returned error can't find the container with id 3fa679615d34813d40c5a7dc5829ccc4a02b4a764a848af569e9d52e7ff276a9
Jul 28 23:23:11.433412 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:11.433371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq58v" event={"ID":"39d5eecd-36b6-4c80-94bf-47cb2888e4f8","Type":"ContainerStarted","Data":"3fa679615d34813d40c5a7dc5829ccc4a02b4a764a848af569e9d52e7ff276a9"}
Jul 28 23:23:11.641818 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:11.641789    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:23:16.459526 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:16.459485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq58v" event={"ID":"39d5eecd-36b6-4c80-94bf-47cb2888e4f8","Type":"ContainerStarted","Data":"0dcfb4d59d4b0ec6cce630fcc68c5703d34ea72778f3640e2aab7b5f0d17cba0"}
Jul 28 23:23:19.472827 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:19.472751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq58v" event={"ID":"39d5eecd-36b6-4c80-94bf-47cb2888e4f8","Type":"ContainerStarted","Data":"8e3786db2f27468dd220c538d8a8b9a0a3d6a06a283e539fa63ec73bf4c814c5"}
Jul 28 23:23:21.809519 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.809485    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zcx2f"]
Jul 28 23:23:21.819232 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.819207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.822023 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.821984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 23:23:21.822150 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.822037    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 23:23:21.822287 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.822273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 23:23:21.823350 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.823107    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ncnk\""
Jul 28 23:23:21.823350 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.823121    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 23:23:21.924814 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.924779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-tls\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.924986 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.924823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-accelerators-collector-config\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.924986 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.924842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-root\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.924986 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.924916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-textfile\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.924986 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.924971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-wtmp\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.925214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.925051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17883e43-b87e-4bbd-8756-3455beed4353-metrics-client-ca\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.925214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.925106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-sys\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.925214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.925130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:21.925214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:21.925153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjc4\" (UniqueName: \"kubernetes.io/projected/17883e43-b87e-4bbd-8756-3455beed4353-kube-api-access-jzjc4\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026341 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-tls\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026508 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-accelerators-collector-config\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026508 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-root\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026508 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-root\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026508 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-textfile\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026731 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:23:22.026504    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 23:23:22.026731 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:23:22.026578    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-tls podName:17883e43-b87e-4bbd-8756-3455beed4353 nodeName:}" failed. No retries permitted until 2026-07-28 23:23:22.526553696 +0000 UTC m=+98.029100206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-tls") pod "node-exporter-zcx2f" (UID: "17883e43-b87e-4bbd-8756-3455beed4353") : secret "node-exporter-tls" not found
Jul 28 23:23:22.026731 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-wtmp\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026731 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-wtmp\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026731 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17883e43-b87e-4bbd-8756-3455beed4353-metrics-client-ca\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026975 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-sys\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026975 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026975 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjc4\" (UniqueName: \"kubernetes.io/projected/17883e43-b87e-4bbd-8756-3455beed4353-kube-api-access-jzjc4\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026975 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-textfile\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.026975 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.026886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17883e43-b87e-4bbd-8756-3455beed4353-sys\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.027193 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.027079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-accelerators-collector-config\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.027431 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.027406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17883e43-b87e-4bbd-8756-3455beed4353-metrics-client-ca\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.029535 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.029511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.037679 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.037653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjc4\" (UniqueName: \"kubernetes.io/projected/17883e43-b87e-4bbd-8756-3455beed4353-kube-api-access-jzjc4\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.351425 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.351395    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cbw8q"
Jul 28 23:23:22.532075 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.532041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-tls\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.534790 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.534764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17883e43-b87e-4bbd-8756-3455beed4353-node-exporter-tls\") pod \"node-exporter-zcx2f\" (UID: \"17883e43-b87e-4bbd-8756-3455beed4353\") " pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.731519 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:22.731479    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zcx2f"
Jul 28 23:23:22.743302 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:22.743265    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17883e43_b87e_4bbd_8756_3455beed4353.slice/crio-919c663de33f0f473941955785cbd2ece9fe11a60fa8b1da616f2645ff823e07 WatchSource:0}: Error finding container 919c663de33f0f473941955785cbd2ece9fe11a60fa8b1da616f2645ff823e07: Status 404 returned error can't find the container with id 919c663de33f0f473941955785cbd2ece9fe11a60fa8b1da616f2645ff823e07
Jul 28 23:23:23.399001 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.398918    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d8b678dc9-6rcqc"
Jul 28 23:23:23.490559 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.490517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jq58v" event={"ID":"39d5eecd-36b6-4c80-94bf-47cb2888e4f8","Type":"ContainerStarted","Data":"e52d3c55f822c88b8069356b9aa67cffc1c466b81c4ef31c9b11485ca39a7f15"}
Jul 28 23:23:23.492036 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.491998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcx2f" event={"ID":"17883e43-b87e-4bbd-8756-3455beed4353","Type":"ContainerStarted","Data":"919c663de33f0f473941955785cbd2ece9fe11a60fa8b1da616f2645ff823e07"}
Jul 28 23:23:23.509083 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.509028    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jq58v" podStartSLOduration=33.521622611 podStartE2EDuration="45.509007007s" totalImagesPullingTime="11.987384396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 23:22:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:23.507917554 +0000 UTC m=+99.010464086" watchObservedRunningTime="2026-07-28 23:23:23.509007007 +0000 UTC m=+99.011553540"
Jul 28 23:23:23.718993 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.718965    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-576699c6dd-2lphd"]
Jul 28 23:23:23.723552 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.723532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.726247 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.726222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 23:23:23.726247 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.726242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 23:23:23.726247 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.726231    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 23:23:23.726453 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.726318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jjbmq\""
Jul 28 23:23:23.726453 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.726257    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 23:23:23.726453 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.726245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f8gokr615u35o\""
Jul 28 23:23:23.726453 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.726250    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 23:23:23.736445 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.736421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-576699c6dd-2lphd"]
Jul 28 23:23:23.742148 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.742269 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.742269 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.742269 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/debb10cc-5bef-44e1-a480-e7969d71a7d7-metrics-client-ca\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.742386 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-grpc-tls\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.742386 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdwh\" (UniqueName: \"kubernetes.io/projected/debb10cc-5bef-44e1-a480-e7969d71a7d7-kube-api-access-pkdwh\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.742386 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-tls\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.742386 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.742383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.842870 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.842834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.842989 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.842884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/debb10cc-5bef-44e1-a480-e7969d71a7d7-metrics-client-ca\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.843061 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.843035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-grpc-tls\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.843180 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.843150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdwh\" (UniqueName: \"kubernetes.io/projected/debb10cc-5bef-44e1-a480-e7969d71a7d7-kube-api-access-pkdwh\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.843265 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.843197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-tls\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.843800 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.843397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.843800 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.843462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.843800 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.843500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.843800 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.843664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/debb10cc-5bef-44e1-a480-e7969d71a7d7-metrics-client-ca\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.846144 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.846087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.846400 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.846333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.846491 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.846461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.846686 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.846665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.846743 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.846700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-grpc-tls\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.847758 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.847736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/debb10cc-5bef-44e1-a480-e7969d71a7d7-secret-thanos-querier-tls\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:23.851208 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:23.851187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdwh\" (UniqueName: \"kubernetes.io/projected/debb10cc-5bef-44e1-a480-e7969d71a7d7-kube-api-access-pkdwh\") pod \"thanos-querier-576699c6dd-2lphd\" (UID: \"debb10cc-5bef-44e1-a480-e7969d71a7d7\") " pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:24.039827 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:24.039728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:24.176896 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:24.176869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-576699c6dd-2lphd"]
Jul 28 23:23:24.179150 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:24.179120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebb10cc_5bef_44e1_a480_e7969d71a7d7.slice/crio-2a4bea526e4be277fae09d49e3868f162490cf02d1214319fa213b2a1de16b36 WatchSource:0}: Error finding container 2a4bea526e4be277fae09d49e3868f162490cf02d1214319fa213b2a1de16b36: Status 404 returned error can't find the container with id 2a4bea526e4be277fae09d49e3868f162490cf02d1214319fa213b2a1de16b36
Jul 28 23:23:24.496610 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:24.496573    2581 generic.go:356] "Generic (PLEG): container finished" podID="17883e43-b87e-4bbd-8756-3455beed4353" containerID="fa5477aa622f7c78a5851261c4ba83c9a8e29ef55924be74c97cb822e29b3c2d" exitCode=0
Jul 28 23:23:24.497104 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:24.496669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcx2f" event={"ID":"17883e43-b87e-4bbd-8756-3455beed4353","Type":"ContainerDied","Data":"fa5477aa622f7c78a5851261c4ba83c9a8e29ef55924be74c97cb822e29b3c2d"}
Jul 28 23:23:24.497870 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:24.497847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" event={"ID":"debb10cc-5bef-44e1-a480-e7969d71a7d7","Type":"ContainerStarted","Data":"2a4bea526e4be277fae09d49e3868f162490cf02d1214319fa213b2a1de16b36"}
Jul 28 23:23:25.504048 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:25.504018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcx2f" event={"ID":"17883e43-b87e-4bbd-8756-3455beed4353","Type":"ContainerStarted","Data":"876e30b6680406babd92a29e09f7edb0e763e38c02537038d84521d2f3a324a7"}
Jul 28 23:23:25.504429 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:25.504092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zcx2f" event={"ID":"17883e43-b87e-4bbd-8756-3455beed4353","Type":"ContainerStarted","Data":"800b2749eb562d6d77d541f2d7f43293272ca2680af911bd4a5fde8cffb8eeba"}
Jul 28 23:23:25.523905 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:25.523853    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zcx2f" podStartSLOduration=3.516361104 podStartE2EDuration="4.523836385s" totalImagesPullingTime="944.013181ms" totalInitContainerRuntime="63.4621ms" isStatefulPod=true podCreationTimestamp="2026-07-28 23:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:25.521964881 +0000 UTC m=+101.024511412" watchObservedRunningTime="2026-07-28 23:23:25.523836385 +0000 UTC m=+101.026382970"
Jul 28 23:23:26.498024 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.497989    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"]
Jul 28 23:23:26.501260 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.501237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.503590 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.503565    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 23:23:26.503743 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.503657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gs9hw\""
Jul 28 23:23:26.510948 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.510915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" event={"ID":"debb10cc-5bef-44e1-a480-e7969d71a7d7","Type":"ContainerStarted","Data":"6e1e834031c30a5b848250e34dd3c07ece13a1546adac5509e2764e915fd97bb"}
Jul 28 23:23:26.511326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.510993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" event={"ID":"debb10cc-5bef-44e1-a480-e7969d71a7d7","Type":"ContainerStarted","Data":"f25ac63c93e985ecfb21307ad06a5ac44d9241a17dc88860dc129ba96a956ae0"}
Jul 28 23:23:26.511326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.511032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" event={"ID":"debb10cc-5bef-44e1-a480-e7969d71a7d7","Type":"ContainerStarted","Data":"94872e30c9f12280b0dca65773f1029ad7124da8a4e6251a786a951bc4140698"}
Jul 28 23:23:26.511790 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.511770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"]
Jul 28 23:23:26.569781 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.569680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ec62df0-dfa1-453c-8987-4d1c7fde37d2-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-t9qrj\" (UID: \"8ec62df0-dfa1-453c-8987-4d1c7fde37d2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.570061 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.570035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn46q\" (UniqueName: \"kubernetes.io/projected/8ec62df0-dfa1-453c-8987-4d1c7fde37d2-kube-api-access-rn46q\") pod \"monitoring-plugin-68b7fc9574-t9qrj\" (UID: \"8ec62df0-dfa1-453c-8987-4d1c7fde37d2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.653190 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.653123    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr" podUID="1c29d603-296e-47fc-8817-19def1f000c4" containerName="registry" containerID="cri-o://e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4" gracePeriod=30
Jul 28 23:23:26.671025 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.670999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn46q\" (UniqueName: \"kubernetes.io/projected/8ec62df0-dfa1-453c-8987-4d1c7fde37d2-kube-api-access-rn46q\") pod \"monitoring-plugin-68b7fc9574-t9qrj\" (UID: \"8ec62df0-dfa1-453c-8987-4d1c7fde37d2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.671112 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.671069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ec62df0-dfa1-453c-8987-4d1c7fde37d2-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-t9qrj\" (UID: \"8ec62df0-dfa1-453c-8987-4d1c7fde37d2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.673368 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.673347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ec62df0-dfa1-453c-8987-4d1c7fde37d2-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-t9qrj\" (UID: \"8ec62df0-dfa1-453c-8987-4d1c7fde37d2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.678296 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.678274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn46q\" (UniqueName: \"kubernetes.io/projected/8ec62df0-dfa1-453c-8987-4d1c7fde37d2-kube-api-access-rn46q\") pod \"monitoring-plugin-68b7fc9574-t9qrj\" (UID: \"8ec62df0-dfa1-453c-8987-4d1c7fde37d2\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.810850 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.810804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:26.898480 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.898315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:23:26.946549 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.946527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"]
Jul 28 23:23:26.962949 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.962925    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"]
Jul 28 23:23:26.963226 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.963211    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c29d603-296e-47fc-8817-19def1f000c4" containerName="registry"
Jul 28 23:23:26.963266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.963229    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c29d603-296e-47fc-8817-19def1f000c4" containerName="registry"
Jul 28 23:23:26.963321 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.963312    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c29d603-296e-47fc-8817-19def1f000c4" containerName="registry"
Jul 28 23:23:26.966427 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.966407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:26.970472 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.970444    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 23:23:26.970613 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.970522    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 23:23:26.970881 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.970860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 23:23:26.970987 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.970963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 23:23:26.971049 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.971033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4w6rs\""
Jul 28 23:23:26.971143 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.971126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 23:23:26.973405 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.973382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.973554 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.973427    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-bound-sa-token\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.973554 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.973464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c29d603-296e-47fc-8817-19def1f000c4-ca-trust-extracted\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.973954 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.973907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hktss\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-kube-api-access-hktss\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.974038 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.974016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-trusted-ca\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.974118 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.974110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-installation-pull-secrets\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.974170 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.974157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-registry-certificates\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.974223 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.974203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-image-registry-private-configuration\") pod \"1c29d603-296e-47fc-8817-19def1f000c4\" (UID: \"1c29d603-296e-47fc-8817-19def1f000c4\") "
Jul 28 23:23:26.974606 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.974578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:23:26.976739 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.975110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:23:26.976739 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.975612    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 23:23:26.977546 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.977497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:23:26.977679 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.977619    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-kube-api-access-hktss" (OuterVolumeSpecName: "kube-api-access-hktss") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "kube-api-access-hktss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:23:26.977859 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.977835    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:23:26.978131 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.978103    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:23:26.978295 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.978281    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:23:26.984846 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.984826    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"]
Jul 28 23:23:26.986214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:26.986189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c29d603-296e-47fc-8817-19def1f000c4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1c29d603-296e-47fc-8817-19def1f000c4" (UID: "1c29d603-296e-47fc-8817-19def1f000c4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:23:27.032387 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.032349    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5db74c54-j2qbn"]
Jul 28 23:23:27.035822 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.035797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.038308 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038285    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 23:23:27.038436 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 23:23:27.038506 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8sgs4\""
Jul 28 23:23:27.038576 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 23:23:27.038653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 23:23:27.038755 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038737    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 23:23:27.038893 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038828    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 23:23:27.038962 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.038876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 23:23:27.044570 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.044540    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5db74c54-j2qbn"]
Jul 28 23:23:27.075047 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.074949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-federate-client-tls\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075047 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-config\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.075047 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-telemeter-client-tls\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075328 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-service-ca\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.075328 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-secret-telemeter-client\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075328 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075328 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-serving-cert\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.075328 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075328 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g77hp\" (UniqueName: \"kubernetes.io/projected/f2161bd0-9deb-4a83-a2e3-bd2929388726-kube-api-access-g77hp\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-metrics-client-ca\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-oauth-config\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-serving-certs-ca-bundle\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ckn\" (UniqueName: \"kubernetes.io/projected/8e3732aa-a669-4ac9-90aa-484e782dc39d-kube-api-access-v7ckn\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-oauth-serving-cert\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075522    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-trusted-ca\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075537    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-installation-pull-secrets\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075555    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c29d603-296e-47fc-8817-19def1f000c4-registry-certificates\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075565    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c29d603-296e-47fc-8817-19def1f000c4-image-registry-private-configuration\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.075592 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075582    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-registry-tls\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.076091 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075599    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-bound-sa-token\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.076091 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075613    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c29d603-296e-47fc-8817-19def1f000c4-ca-trust-extracted\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.076091 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.075644    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktss\" (UniqueName: \"kubernetes.io/projected/1c29d603-296e-47fc-8817-19def1f000c4-kube-api-access-hktss\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:23:27.176172 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77hp\" (UniqueName: \"kubernetes.io/projected/f2161bd0-9deb-4a83-a2e3-bd2929388726-kube-api-access-g77hp\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-metrics-client-ca\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-oauth-config\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.176392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-serving-certs-ca-bundle\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ckn\" (UniqueName: \"kubernetes.io/projected/8e3732aa-a669-4ac9-90aa-484e782dc39d-kube-api-access-v7ckn\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.176392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-oauth-serving-cert\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.176392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-federate-client-tls\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-config\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.176810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-telemeter-client-tls\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-service-ca\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.176810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-secret-telemeter-client\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.176810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.176565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-serving-cert\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.177096 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.177063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-metrics-client-ca\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.177472 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.177187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.177472 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.177243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-oauth-serving-cert\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.177659 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.177501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-service-ca\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.177747 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.177720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2161bd0-9deb-4a83-a2e3-bd2929388726-serving-certs-ca-bundle\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.177888 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.177831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-config\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.180089 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.180044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-serving-cert\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.180192 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.180098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-telemeter-client-tls\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.180192 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.180159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-oauth-config\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.180367 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.180346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-secret-telemeter-client\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.180446 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.180376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-federate-client-tls\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.180486 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.180453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2161bd0-9deb-4a83-a2e3-bd2929388726-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.184170 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.184152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77hp\" (UniqueName: \"kubernetes.io/projected/f2161bd0-9deb-4a83-a2e3-bd2929388726-kube-api-access-g77hp\") pod \"telemeter-client-5ff44949c6-hmkfn\" (UID: \"f2161bd0-9deb-4a83-a2e3-bd2929388726\") " pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.184677 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.184655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ckn\" (UniqueName: \"kubernetes.io/projected/8e3732aa-a669-4ac9-90aa-484e782dc39d-kube-api-access-v7ckn\") pod \"console-b5db74c54-j2qbn\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") " pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.277834 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.277794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"
Jul 28 23:23:27.347804 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.347768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:27.464279 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.463097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5ff44949c6-hmkfn"]
Jul 28 23:23:27.499059 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.499025    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5db74c54-j2qbn"]
Jul 28 23:23:27.503780 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:27.503750    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3732aa_a669_4ac9_90aa_484e782dc39d.slice/crio-daa4416b8a310d63997855cb8a681992e49f348d6cba7d9d98628909c8194ff3 WatchSource:0}: Error finding container daa4416b8a310d63997855cb8a681992e49f348d6cba7d9d98628909c8194ff3: Status 404 returned error can't find the container with id daa4416b8a310d63997855cb8a681992e49f348d6cba7d9d98628909c8194ff3
Jul 28 23:23:27.517789 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.517760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" event={"ID":"debb10cc-5bef-44e1-a480-e7969d71a7d7","Type":"ContainerStarted","Data":"5143ad20228bf4273983ff372a48dd202c454448ec3eab0f9596acdebcef576a"}
Jul 28 23:23:27.518133 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.517836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" event={"ID":"debb10cc-5bef-44e1-a480-e7969d71a7d7","Type":"ContainerStarted","Data":"f24f95df02833ea2952265a537e94a393c63757d53e8f1918e16d481d96e5ddb"}
Jul 28 23:23:27.519494 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.519471    2581 generic.go:356] "Generic (PLEG): container finished" podID="1c29d603-296e-47fc-8817-19def1f000c4" containerID="e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4" exitCode=0
Jul 28 23:23:27.519590 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.519538    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr"
Jul 28 23:23:27.519590 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.519553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr" event={"ID":"1c29d603-296e-47fc-8817-19def1f000c4","Type":"ContainerDied","Data":"e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4"}
Jul 28 23:23:27.519682 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.519605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7869b868ff-lcdtr" event={"ID":"1c29d603-296e-47fc-8817-19def1f000c4","Type":"ContainerDied","Data":"e0d9b38156809f2c0c227f63991947d17bc2a2419ee517f4f6cb9faf57ef14fc"}
Jul 28 23:23:27.519682 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.519648    2581 scope.go:117] "RemoveContainer" containerID="e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4"
Jul 28 23:23:27.521357 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.521297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn" event={"ID":"f2161bd0-9deb-4a83-a2e3-bd2929388726","Type":"ContainerStarted","Data":"bcf25c9fe68df975d58a4dd52149df2abb1808f6a2425ebddf10629d58ad3ac1"}
Jul 28 23:23:27.522747 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.522712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5db74c54-j2qbn" event={"ID":"8e3732aa-a669-4ac9-90aa-484e782dc39d","Type":"ContainerStarted","Data":"daa4416b8a310d63997855cb8a681992e49f348d6cba7d9d98628909c8194ff3"}
Jul 28 23:23:27.523839 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.523816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj" event={"ID":"8ec62df0-dfa1-453c-8987-4d1c7fde37d2","Type":"ContainerStarted","Data":"e0ee53482abb30cb59f9c19814fc54e1dc6b1a42e564cc6cf137944dd6ef7de3"}
Jul 28 23:23:27.528095 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.528081    2581 scope.go:117] "RemoveContainer" containerID="e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4"
Jul 28 23:23:27.528349 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:23:27.528323    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4\": container with ID starting with e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4 not found: ID does not exist" containerID="e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4"
Jul 28 23:23:27.528401 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.528362    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4"} err="failed to get container status \"e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4\": rpc error: code = NotFound desc = could not find container \"e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4\": container with ID starting with e34dd99a0d08f0877d9387485a68c7839538c82328c233060d145ef746b53ef4 not found: ID does not exist"
Jul 28 23:23:27.541216 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.541189    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7869b868ff-lcdtr"]
Jul 28 23:23:27.544511 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.544489    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7869b868ff-lcdtr"]
Jul 28 23:23:27.970959 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.970918    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:23:27.974967 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.974942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:27.977552 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.977527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 23:23:27.977693 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.977541    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gmn57\""
Jul 28 23:23:27.977746 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.977548    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 23:23:27.977979 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.977955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 23:23:27.978798 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.978779    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 23:23:27.978798 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.978788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 23:23:27.978944 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.978782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 23:23:27.979912 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.979872    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 23:23:27.980013 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.979917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 23:23:27.980013 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.979923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 23:23:27.980537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.980305    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 23:23:27.980537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.980341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 23:23:27.980537 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.980346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 23:23:27.980881 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.980750    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fr9h71mveglbf\""
Jul 28 23:23:27.980881 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.980771    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 23:23:27.981057 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.980905    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 23:23:27.981506 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.981383    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 23:23:27.988735 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:27.988711    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:23:28.085349 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085536 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085536 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085536 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwwvn\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-kube-api-access-dwwvn\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.085757 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.085936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086055 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.086039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086382 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.086072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086382 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.086093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.086382 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.086118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.186764 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.186936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.186936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.186936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.186936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.186936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwwvn\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-kube-api-access-dwwvn\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.186936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.186974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.187908 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.187310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.188461 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.188439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.189637 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.189602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192064 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.190337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192064 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.190360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192064 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.190896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192064 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.191220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192064 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.191509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192064 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.191985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192421 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.192318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.192890 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.192830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.196490 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.196431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.196584 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.196523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.198000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.197015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.198000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.197069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.198000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.197408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.198000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.197580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.198000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.197964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.199192 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.199142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.200062 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.200023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwwvn\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-kube-api-access-dwwvn\") pod \"prometheus-k8s-0\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.287023 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.286938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:28.539683 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.538964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" event={"ID":"debb10cc-5bef-44e1-a480-e7969d71a7d7","Type":"ContainerStarted","Data":"64a6068d7c596ea8d8d953f3c7f219ff60c7e2a1589dbfa0dc739c755c5f3b21"}
Jul 28 23:23:28.540213 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.540139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:28.570867 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.567255    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd" podStartSLOduration=2.623455121 podStartE2EDuration="5.567230864s" totalImagesPullingTime="2.943775743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:23:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:28.563692347 +0000 UTC m=+104.066238880" watchObservedRunningTime="2026-07-28 23:23:28.567230864 +0000 UTC m=+104.069777396"
Jul 28 23:23:28.716654 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.716607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:23:28.725332 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:28.724008    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd0ce57_e07d_4017_8e8b_b1daa9bf1d25.slice/crio-b23169d1e23fa9e99c59ece028148dff92749a0a40777b85a44b9e7f99dafb10 WatchSource:0}: Error finding container b23169d1e23fa9e99c59ece028148dff92749a0a40777b85a44b9e7f99dafb10: Status 404 returned error can't find the container with id b23169d1e23fa9e99c59ece028148dff92749a0a40777b85a44b9e7f99dafb10
Jul 28 23:23:28.982458 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:28.982421    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c29d603-296e-47fc-8817-19def1f000c4" path="/var/lib/kubelet/pods/1c29d603-296e-47fc-8817-19def1f000c4/volumes"
Jul 28 23:23:29.549593 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:29.549006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj" event={"ID":"8ec62df0-dfa1-453c-8987-4d1c7fde37d2","Type":"ContainerStarted","Data":"d436549a83dcad4b979507308cd9c7694123b11ee2c1b8ffdbab893ff4962485"}
Jul 28 23:23:29.549593 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:29.549538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:29.551060 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:29.551028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"b23169d1e23fa9e99c59ece028148dff92749a0a40777b85a44b9e7f99dafb10"}
Jul 28 23:23:29.555918 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:29.555890    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj"
Jul 28 23:23:29.563902 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:29.563849    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-t9qrj" podStartSLOduration=1.922706556 podStartE2EDuration="3.56383016s" totalImagesPullingTime="1.641123604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:29.563201614 +0000 UTC m=+105.065748147" watchObservedRunningTime="2026-07-28 23:23:29.56383016 +0000 UTC m=+105.066376692"
Jul 28 23:23:31.560195 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:31.560100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5db74c54-j2qbn" event={"ID":"8e3732aa-a669-4ac9-90aa-484e782dc39d","Type":"ContainerStarted","Data":"d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5"}
Jul 28 23:23:31.561672 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:31.561620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"}
Jul 28 23:23:31.563797 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:31.563772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn" event={"ID":"f2161bd0-9deb-4a83-a2e3-bd2929388726","Type":"ContainerStarted","Data":"29d14694d0560c97b2e8929d7de495b470f6e29d8fa7c70e2845dbc5288368ad"}
Jul 28 23:23:31.582974 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:31.579051    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b5db74c54-j2qbn" podStartSLOduration=0.647543596 podStartE2EDuration="4.579030645s" totalImagesPullingTime="3.931487049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:31.574766731 +0000 UTC m=+107.077313260" watchObservedRunningTime="2026-07-28 23:23:31.579030645 +0000 UTC m=+107.081577177"
Jul 28 23:23:32.568989 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:32.568954    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e" exitCode=0
Jul 28 23:23:32.569411 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:32.569040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"}
Jul 28 23:23:32.571258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:32.571234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn" event={"ID":"f2161bd0-9deb-4a83-a2e3-bd2929388726","Type":"ContainerStarted","Data":"908a9526461e68a4fb057df5468ba4ea9646123cf1fae619c282003958638b34"}
Jul 28 23:23:32.571340 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:32.571282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn" event={"ID":"f2161bd0-9deb-4a83-a2e3-bd2929388726","Type":"ContainerStarted","Data":"2ecbca7c34ef38d71fb3dda063e90dba271f2cf0922df009e63cc19525badba4"}
Jul 28 23:23:32.612423 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:32.612378    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5ff44949c6-hmkfn" podStartSLOduration=2.9214780080000002 podStartE2EDuration="6.612363174s" totalImagesPullingTime="3.690885166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:32.610498421 +0000 UTC m=+108.113044952" watchObservedRunningTime="2026-07-28 23:23:32.612363174 +0000 UTC m=+108.114909705"
Jul 28 23:23:34.557679 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:34.557267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-576699c6dd-2lphd"
Jul 28 23:23:36.593153 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:36.593118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"}
Jul 28 23:23:36.593653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:36.593213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"}
Jul 28 23:23:36.593653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:36.593240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"}
Jul 28 23:23:36.593653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:36.593265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"}
Jul 28 23:23:36.593653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:36.593288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"}
Jul 28 23:23:36.593653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:36.593311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerStarted","Data":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"}
Jul 28 23:23:36.625549 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:36.625501    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.765177945 podStartE2EDuration="9.625484621s" totalImagesPullingTime="5.829677327s" totalInitContainerRuntime="30.629349ms" isStatefulPod=false podCreationTimestamp="2026-07-28 23:23:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:36.623802715 +0000 UTC m=+112.126349257" watchObservedRunningTime="2026-07-28 23:23:36.625484621 +0000 UTC m=+112.128031154"
Jul 28 23:23:37.348895 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:37.348854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:37.349095 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:37.348920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:37.353800 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:37.353776    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:37.600688 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:37.600601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:23:38.287487 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:38.287445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:23:44.627327 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:44.627290    2581 generic.go:356] "Generic (PLEG): container finished" podID="619deb1c-42ab-4f7f-b489-ff4e04e0fd00" containerID="c60c65ff9da68dfea076065f112005c805020f3a1e0088b5412709afc3ff6070" exitCode=0
Jul 28 23:23:44.627715 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:44.627363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh" event={"ID":"619deb1c-42ab-4f7f-b489-ff4e04e0fd00","Type":"ContainerDied","Data":"c60c65ff9da68dfea076065f112005c805020f3a1e0088b5412709afc3ff6070"}
Jul 28 23:23:44.627760 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:44.627722    2581 scope.go:117] "RemoveContainer" containerID="c60c65ff9da68dfea076065f112005c805020f3a1e0088b5412709afc3ff6070"
Jul 28 23:23:45.635942 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:45.635907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-ltnqh" event={"ID":"619deb1c-42ab-4f7f-b489-ff4e04e0fd00","Type":"ContainerStarted","Data":"1bb3589bf0bf877772848dfde54e6d3630f1287e82441fb94c6d30571144f925"}
Jul 28 23:23:47.810102 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:47.810056    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5db74c54-j2qbn"]
Jul 28 23:23:54.833575 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:54.833541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:23:54.835824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:54.835802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6-metrics-certs\") pod \"network-metrics-daemon-m968q\" (UID: \"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6\") " pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:23:55.087168 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:55.087093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plrk7\""
Jul 28 23:23:55.095099 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:55.095080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m968q"
Jul 28 23:23:55.269444 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:55.269414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m968q"]
Jul 28 23:23:55.273729 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:23:55.273691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a6c5f_b4da_42c5_9fd6_338c0a275ef6.slice/crio-62d5ba42fcb1cb29a8d196d548f3f893ce61740e3868cd82ce54af10050a7606 WatchSource:0}: Error finding container 62d5ba42fcb1cb29a8d196d548f3f893ce61740e3868cd82ce54af10050a7606: Status 404 returned error can't find the container with id 62d5ba42fcb1cb29a8d196d548f3f893ce61740e3868cd82ce54af10050a7606
Jul 28 23:23:55.679789 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:55.679754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m968q" event={"ID":"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6","Type":"ContainerStarted","Data":"62d5ba42fcb1cb29a8d196d548f3f893ce61740e3868cd82ce54af10050a7606"}
Jul 28 23:23:57.688923 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:57.688886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m968q" event={"ID":"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6","Type":"ContainerStarted","Data":"2081b55125fcec2b84a71758ee9c4f1054dfc0bb6451cbf4746ad7f37dd162e1"}
Jul 28 23:23:57.689415 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:57.688943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m968q" event={"ID":"0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6","Type":"ContainerStarted","Data":"6e22252a4aa8cc5be5ff4a659ca966ebf1e8d72cf8948b455725d94d8b717c9d"}
Jul 28 23:23:57.704620 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:23:57.704574    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m968q" podStartSLOduration=131.197392009 podStartE2EDuration="2m12.704557655s" totalImagesPullingTime="1.507165646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:23:57.702477032 +0000 UTC m=+133.205023557" watchObservedRunningTime="2026-07-28 23:23:57.704557655 +0000 UTC m=+133.207104179"
Jul 28 23:24:09.747708 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:09.747674    2581 generic.go:356] "Generic (PLEG): container finished" podID="71381373-e286-4cf0-8f0c-5048409119e9" containerID="414c58bb6b38a72cbe22b79dd66312b38107ef52ff61a7530054250691d94b36" exitCode=0
Jul 28 23:24:09.748141 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:09.747751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm" event={"ID":"71381373-e286-4cf0-8f0c-5048409119e9","Type":"ContainerDied","Data":"414c58bb6b38a72cbe22b79dd66312b38107ef52ff61a7530054250691d94b36"}
Jul 28 23:24:09.748141 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:09.748114    2581 scope.go:117] "RemoveContainer" containerID="414c58bb6b38a72cbe22b79dd66312b38107ef52ff61a7530054250691d94b36"
Jul 28 23:24:10.754776 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:10.754738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm" event={"ID":"71381373-e286-4cf0-8f0c-5048409119e9","Type":"ContainerStarted","Data":"3dc497e2789e706f36fd0f84f4a8d39d35a0baf93c7fbaa40cee2c369ccf4e65"}
Jul 28 23:24:12.844608 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:12.844540    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b5db74c54-j2qbn" podUID="8e3732aa-a669-4ac9-90aa-484e782dc39d" containerName="console" containerID="cri-o://d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5" gracePeriod=15
Jul 28 23:24:13.124190 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.124158    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:24:13.200960 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.200919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7ckn\" (UniqueName: \"kubernetes.io/projected/8e3732aa-a669-4ac9-90aa-484e782dc39d-kube-api-access-v7ckn\") pod \"8e3732aa-a669-4ac9-90aa-484e782dc39d\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") "
Jul 28 23:24:13.201238 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.201218    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-oauth-serving-cert\") pod \"8e3732aa-a669-4ac9-90aa-484e782dc39d\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") "
Jul 28 23:24:13.201365 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.201350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-oauth-config\") pod \"8e3732aa-a669-4ac9-90aa-484e782dc39d\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") "
Jul 28 23:24:13.201525 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.201503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-config\") pod \"8e3732aa-a669-4ac9-90aa-484e782dc39d\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") "
Jul 28 23:24:13.201659 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.201645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-service-ca\") pod \"8e3732aa-a669-4ac9-90aa-484e782dc39d\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") "
Jul 28 23:24:13.201837 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.201821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-serving-cert\") pod \"8e3732aa-a669-4ac9-90aa-484e782dc39d\" (UID: \"8e3732aa-a669-4ac9-90aa-484e782dc39d\") "
Jul 28 23:24:13.202465 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.202061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8e3732aa-a669-4ac9-90aa-484e782dc39d" (UID: "8e3732aa-a669-4ac9-90aa-484e782dc39d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:13.202465 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.202187    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-oauth-serving-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:13.203292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.203019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-config" (OuterVolumeSpecName: "console-config") pod "8e3732aa-a669-4ac9-90aa-484e782dc39d" (UID: "8e3732aa-a669-4ac9-90aa-484e782dc39d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:13.204796 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.203926    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-service-ca" (OuterVolumeSpecName: "service-ca") pod "8e3732aa-a669-4ac9-90aa-484e782dc39d" (UID: "8e3732aa-a669-4ac9-90aa-484e782dc39d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:13.208109 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.208077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8e3732aa-a669-4ac9-90aa-484e782dc39d" (UID: "8e3732aa-a669-4ac9-90aa-484e782dc39d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:13.209369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.209345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8e3732aa-a669-4ac9-90aa-484e782dc39d" (UID: "8e3732aa-a669-4ac9-90aa-484e782dc39d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:13.209487 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.209468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3732aa-a669-4ac9-90aa-484e782dc39d-kube-api-access-v7ckn" (OuterVolumeSpecName: "kube-api-access-v7ckn") pod "8e3732aa-a669-4ac9-90aa-484e782dc39d" (UID: "8e3732aa-a669-4ac9-90aa-484e782dc39d"). InnerVolumeSpecName "kube-api-access-v7ckn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:24:13.303747 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.303707    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-serving-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:13.303747 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.303750    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7ckn\" (UniqueName: \"kubernetes.io/projected/8e3732aa-a669-4ac9-90aa-484e782dc39d-kube-api-access-v7ckn\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:13.304025 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.303768    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-oauth-config\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:13.304025 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.303784    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-console-config\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:13.304025 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.303799    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e3732aa-a669-4ac9-90aa-484e782dc39d-service-ca\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:13.772977 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.772946    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e3732aa-a669-4ac9-90aa-484e782dc39d" containerID="d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5" exitCode=0
Jul 28 23:24:13.773146 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.772983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5db74c54-j2qbn" event={"ID":"8e3732aa-a669-4ac9-90aa-484e782dc39d","Type":"ContainerDied","Data":"d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5"}
Jul 28 23:24:13.773146 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.772994    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5db74c54-j2qbn"
Jul 28 23:24:13.773146 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.773020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5db74c54-j2qbn" event={"ID":"8e3732aa-a669-4ac9-90aa-484e782dc39d","Type":"ContainerDied","Data":"daa4416b8a310d63997855cb8a681992e49f348d6cba7d9d98628909c8194ff3"}
Jul 28 23:24:13.773146 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.773040    2581 scope.go:117] "RemoveContainer" containerID="d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5"
Jul 28 23:24:13.781316 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.781299    2581 scope.go:117] "RemoveContainer" containerID="d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5"
Jul 28 23:24:13.781569 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:13.781552    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5\": container with ID starting with d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5 not found: ID does not exist" containerID="d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5"
Jul 28 23:24:13.781650 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.781580    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5"} err="failed to get container status \"d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5\": rpc error: code = NotFound desc = could not find container \"d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5\": container with ID starting with d62de2a368feae5eef45e20b363ec3d67b9540591993da8400ea260ff7b6d9f5 not found: ID does not exist"
Jul 28 23:24:13.793604 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.793576    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5db74c54-j2qbn"]
Jul 28 23:24:13.794973 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:13.794954    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b5db74c54-j2qbn"]
Jul 28 23:24:14.982522 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:14.982483    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3732aa-a669-4ac9-90aa-484e782dc39d" path="/var/lib/kubelet/pods/8e3732aa-a669-4ac9-90aa-484e782dc39d/volumes"
Jul 28 23:24:28.287915 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:28.287879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:28.307060 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:28.307026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:28.852505 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:28.852478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.298147 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.298113    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:24:46.298674 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.298387    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="prometheus" containerID="cri-o://46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22" gracePeriod=600
Jul 28 23:24:46.298674 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.298436    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-web" containerID="cri-o://d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1" gracePeriod=600
Jul 28 23:24:46.298674 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.298437    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy" containerID="cri-o://e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd" gracePeriod=600
Jul 28 23:24:46.298674 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.298441    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="thanos-sidecar" containerID="cri-o://5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477" gracePeriod=600
Jul 28 23:24:46.298674 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.298492    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="config-reloader" containerID="cri-o://02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c" gracePeriod=600
Jul 28 23:24:46.298674 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.298523    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946" gracePeriod=600
Jul 28 23:24:46.570116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.570085    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.586287 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-metrics-client-certs\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586437 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-kubelet-serving-ca-bundle\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586437 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586437 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-trusted-ca-bundle\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586437 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-kube-rbac-proxy\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586437 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586383    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-metrics-client-ca\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586437 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-db\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586437 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config-out\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-2\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwwvn\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-kube-api-access-dwwvn\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586593    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-0\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-1\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-thanos-prometheus-http-client-file\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-tls-assets\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-grpc-tls\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.586824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586802    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-web-config\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-serving-certs-ca-bundle\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.586875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-tls\") pod \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\" (UID: \"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25\") "
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587255    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587274    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587288    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.587307 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587305    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-configmap-metrics-client-ca\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.587804 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587703    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:46.587952 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.587923    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:24:46.589116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.588543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:46.589257 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.589220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.589408 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.589224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:24:46.590062 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.590027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.590656 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.590616    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.590901 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.590872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.591110 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.591090    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.591511 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.591485    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.591598 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.591512    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.591598 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.591522    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-kube-api-access-dwwvn" (OuterVolumeSpecName: "kube-api-access-dwwvn") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "kube-api-access-dwwvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:24:46.591598 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.591529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config-out" (OuterVolumeSpecName: "config-out") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:24:46.591740 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.591621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:24:46.591775 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.591735    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config" (OuterVolumeSpecName: "config") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.603085 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.603049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-web-config" (OuterVolumeSpecName: "web-config") pod "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" (UID: "4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:24:46.693928 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693894    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-kube-rbac-proxy\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.693928 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693923    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-db\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.693928 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693933    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config-out\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693942    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-config\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693952    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693962    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693970    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwwvn\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-kube-api-access-dwwvn\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693978    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693986    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.693994    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.694003    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-tls-assets\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.694011    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-grpc-tls\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.694018    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-web-config\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.694026    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.694034    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-metrics-client-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.694254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.694042    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:24:46.912082 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912041    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946" exitCode=0
Jul 28 23:24:46.912082 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912076    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd" exitCode=0
Jul 28 23:24:46.912082 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912085    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1" exitCode=0
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912093    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477" exitCode=0
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912101    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c" exitCode=0
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912110    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22" exitCode=0
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"}
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912145    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"}
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"}
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"}
Jul 28 23:24:46.912298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912294    2581 scope.go:117] "RemoveContainer" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.912579 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"}
Jul 28 23:24:46.912579 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"}
Jul 28 23:24:46.912579 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.912357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25","Type":"ContainerDied","Data":"b23169d1e23fa9e99c59ece028148dff92749a0a40777b85a44b9e7f99dafb10"}
Jul 28 23:24:46.920287 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.920270    2581 scope.go:117] "RemoveContainer" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.927336 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.927312    2581 scope.go:117] "RemoveContainer" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.934919 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.934900    2581 scope.go:117] "RemoveContainer" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.935391 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.935372    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:24:46.939745 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.939726    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:24:46.942041 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.942025    2581 scope.go:117] "RemoveContainer" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.948335 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.948319    2581 scope.go:117] "RemoveContainer" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.955374 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.955360    2581 scope.go:117] "RemoveContainer" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.961455 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961433    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:24:46.961902 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961883    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3732aa-a669-4ac9-90aa-484e782dc39d" containerName="console"
Jul 28 23:24:46.961953 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961905    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3732aa-a669-4ac9-90aa-484e782dc39d" containerName="console"
Jul 28 23:24:46.961953 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961925    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="config-reloader"
Jul 28 23:24:46.961953 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961934    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="config-reloader"
Jul 28 23:24:46.961953 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961944    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="prometheus"
Jul 28 23:24:46.961953 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961953    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="prometheus"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961963    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961971    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.961994    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-thanos"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962004    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-thanos"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962020    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-web"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962028    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-web"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962040    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="thanos-sidecar"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962048    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="thanos-sidecar"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962057    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="init-config-reloader"
Jul 28 23:24:46.962097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962066    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="init-config-reloader"
Jul 28 23:24:46.962369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962157    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3732aa-a669-4ac9-90aa-484e782dc39d" containerName="console"
Jul 28 23:24:46.962369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962170    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-thanos"
Jul 28 23:24:46.962369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962179    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="prometheus"
Jul 28 23:24:46.962369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962187    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy"
Jul 28 23:24:46.962369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962200    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="config-reloader"
Jul 28 23:24:46.962369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962209    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="kube-rbac-proxy-web"
Jul 28 23:24:46.962369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.962218    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" containerName="thanos-sidecar"
Jul 28 23:24:46.963281 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.963259    2581 scope.go:117] "RemoveContainer" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.963530 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:46.963511    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": container with ID starting with 2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946 not found: ID does not exist" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.963598 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.963542    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"} err="failed to get container status \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": rpc error: code = NotFound desc = could not find container \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": container with ID starting with 2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946 not found: ID does not exist"
Jul 28 23:24:46.963598 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.963568    2581 scope.go:117] "RemoveContainer" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.963857 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:46.963839    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": container with ID starting with e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd not found: ID does not exist" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.963900 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.963864    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"} err="failed to get container status \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": rpc error: code = NotFound desc = could not find container \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": container with ID starting with e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd not found: ID does not exist"
Jul 28 23:24:46.963900 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.963881    2581 scope.go:117] "RemoveContainer" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.964094 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:46.964075    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": container with ID starting with d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1 not found: ID does not exist" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.964158 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964101    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"} err="failed to get container status \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": rpc error: code = NotFound desc = could not find container \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": container with ID starting with d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1 not found: ID does not exist"
Jul 28 23:24:46.964158 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964121    2581 scope.go:117] "RemoveContainer" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.964359 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:46.964343    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": container with ID starting with 5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477 not found: ID does not exist" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.964400 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964362    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"} err="failed to get container status \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": rpc error: code = NotFound desc = could not find container \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": container with ID starting with 5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477 not found: ID does not exist"
Jul 28 23:24:46.964400 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964374    2581 scope.go:117] "RemoveContainer" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.964570 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:46.964554    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": container with ID starting with 02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c not found: ID does not exist" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.964647 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964576    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"} err="failed to get container status \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": rpc error: code = NotFound desc = could not find container \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": container with ID starting with 02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c not found: ID does not exist"
Jul 28 23:24:46.964647 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964594    2581 scope.go:117] "RemoveContainer" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.964849 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:46.964830    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": container with ID starting with 46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22 not found: ID does not exist" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.964925 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964852    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"} err="failed to get container status \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": rpc error: code = NotFound desc = could not find container \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": container with ID starting with 46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22 not found: ID does not exist"
Jul 28 23:24:46.964925 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.964868    2581 scope.go:117] "RemoveContainer" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.965074 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:24:46.965060    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": container with ID starting with 25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e not found: ID does not exist" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.965116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965077    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"} err="failed to get container status \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": rpc error: code = NotFound desc = could not find container \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": container with ID starting with 25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e not found: ID does not exist"
Jul 28 23:24:46.965116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965089    2581 scope.go:117] "RemoveContainer" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.965330 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965311    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"} err="failed to get container status \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": rpc error: code = NotFound desc = could not find container \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": container with ID starting with 2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946 not found: ID does not exist"
Jul 28 23:24:46.965381 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965331    2581 scope.go:117] "RemoveContainer" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.965542 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965526    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"} err="failed to get container status \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": rpc error: code = NotFound desc = could not find container \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": container with ID starting with e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd not found: ID does not exist"
Jul 28 23:24:46.965583 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965543    2581 scope.go:117] "RemoveContainer" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.965758 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965740    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"} err="failed to get container status \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": rpc error: code = NotFound desc = could not find container \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": container with ID starting with d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1 not found: ID does not exist"
Jul 28 23:24:46.965818 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965758    2581 scope.go:117] "RemoveContainer" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.965991 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965970    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"} err="failed to get container status \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": rpc error: code = NotFound desc = could not find container \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": container with ID starting with 5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477 not found: ID does not exist"
Jul 28 23:24:46.965991 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.965991    2581 scope.go:117] "RemoveContainer" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.966212 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966196    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"} err="failed to get container status \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": rpc error: code = NotFound desc = could not find container \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": container with ID starting with 02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c not found: ID does not exist"
Jul 28 23:24:46.966282 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966212    2581 scope.go:117] "RemoveContainer" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.966406 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966389    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"} err="failed to get container status \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": rpc error: code = NotFound desc = could not find container \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": container with ID starting with 46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22 not found: ID does not exist"
Jul 28 23:24:46.966455 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966408    2581 scope.go:117] "RemoveContainer" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.966603 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966581    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"} err="failed to get container status \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": rpc error: code = NotFound desc = could not find container \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": container with ID starting with 25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e not found: ID does not exist"
Jul 28 23:24:46.966698 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966605    2581 scope.go:117] "RemoveContainer" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.966821 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966801    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"} err="failed to get container status \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": rpc error: code = NotFound desc = could not find container \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": container with ID starting with 2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946 not found: ID does not exist"
Jul 28 23:24:46.966862 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966823    2581 scope.go:117] "RemoveContainer" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.966994 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966978    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"} err="failed to get container status \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": rpc error: code = NotFound desc = could not find container \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": container with ID starting with e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd not found: ID does not exist"
Jul 28 23:24:46.967033 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.966996    2581 scope.go:117] "RemoveContainer" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.967209 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967194    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"} err="failed to get container status \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": rpc error: code = NotFound desc = could not find container \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": container with ID starting with d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1 not found: ID does not exist"
Jul 28 23:24:46.967244 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967210    2581 scope.go:117] "RemoveContainer" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.967415 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967395    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"} err="failed to get container status \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": rpc error: code = NotFound desc = could not find container \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": container with ID starting with 5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477 not found: ID does not exist"
Jul 28 23:24:46.967415 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967414    2581 scope.go:117] "RemoveContainer" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.967610 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967595    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"} err="failed to get container status \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": rpc error: code = NotFound desc = could not find container \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": container with ID starting with 02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c not found: ID does not exist"
Jul 28 23:24:46.967743 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967612    2581 scope.go:117] "RemoveContainer" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.967908 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967888    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"} err="failed to get container status \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": rpc error: code = NotFound desc = could not find container \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": container with ID starting with 46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22 not found: ID does not exist"
Jul 28 23:24:46.967955 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967909    2581 scope.go:117] "RemoveContainer" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.967955 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.967901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.968388 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.968175    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"} err="failed to get container status \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": rpc error: code = NotFound desc = could not find container \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": container with ID starting with 25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e not found: ID does not exist"
Jul 28 23:24:46.968388 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.968209    2581 scope.go:117] "RemoveContainer" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.968532 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.968504    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"} err="failed to get container status \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": rpc error: code = NotFound desc = could not find container \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": container with ID starting with 2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946 not found: ID does not exist"
Jul 28 23:24:46.968532 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.968528    2581 scope.go:117] "RemoveContainer" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.968810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.968786    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"} err="failed to get container status \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": rpc error: code = NotFound desc = could not find container \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": container with ID starting with e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd not found: ID does not exist"
Jul 28 23:24:46.968882 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.968812    2581 scope.go:117] "RemoveContainer" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.969043 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969024    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"} err="failed to get container status \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": rpc error: code = NotFound desc = could not find container \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": container with ID starting with d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1 not found: ID does not exist"
Jul 28 23:24:46.969106 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969046    2581 scope.go:117] "RemoveContainer" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.969312 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969285    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"} err="failed to get container status \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": rpc error: code = NotFound desc = could not find container \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": container with ID starting with 5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477 not found: ID does not exist"
Jul 28 23:24:46.969352 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969315    2581 scope.go:117] "RemoveContainer" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.969576 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969529    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"} err="failed to get container status \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": rpc error: code = NotFound desc = could not find container \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": container with ID starting with 02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c not found: ID does not exist"
Jul 28 23:24:46.969576 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969547    2581 scope.go:117] "RemoveContainer" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.969763 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969744    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"} err="failed to get container status \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": rpc error: code = NotFound desc = could not find container \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": container with ID starting with 46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22 not found: ID does not exist"
Jul 28 23:24:46.969859 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969765    2581 scope.go:117] "RemoveContainer" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.969976 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969955    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"} err="failed to get container status \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": rpc error: code = NotFound desc = could not find container \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": container with ID starting with 25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e not found: ID does not exist"
Jul 28 23:24:46.970016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.969978    2581 scope.go:117] "RemoveContainer" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.970220 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970200    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"} err="failed to get container status \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": rpc error: code = NotFound desc = could not find container \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": container with ID starting with 2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946 not found: ID does not exist"
Jul 28 23:24:46.970277 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970221    2581 scope.go:117] "RemoveContainer" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.970443 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970426    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"} err="failed to get container status \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": rpc error: code = NotFound desc = could not find container \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": container with ID starting with e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd not found: ID does not exist"
Jul 28 23:24:46.970492 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970444    2581 scope.go:117] "RemoveContainer" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.970615 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970599    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fr9h71mveglbf\""
Jul 28 23:24:46.970718 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970686    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"} err="failed to get container status \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": rpc error: code = NotFound desc = could not find container \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": container with ID starting with d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1 not found: ID does not exist"
Jul 28 23:24:46.970718 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970707    2581 scope.go:117] "RemoveContainer" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.970831 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gmn57\""
Jul 28 23:24:46.970873 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970843    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 23:24:46.970873 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 23:24:46.970932 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 23:24:46.970963 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970818    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 23:24:46.970963 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970862    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 23:24:46.971025 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.970998    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 23:24:46.971221 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971197    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"} err="failed to get container status \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": rpc error: code = NotFound desc = could not find container \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": container with ID starting with 5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477 not found: ID does not exist"
Jul 28 23:24:46.971306 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971223    2581 scope.go:117] "RemoveContainer" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.971306 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971227    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 23:24:46.971306 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971286    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 23:24:46.971306 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 23:24:46.971529 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"} err="failed to get container status \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": rpc error: code = NotFound desc = could not find container \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": container with ID starting with 02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c not found: ID does not exist"
Jul 28 23:24:46.971571 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971530    2581 scope.go:117] "RemoveContainer" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.971791 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971770    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"} err="failed to get container status \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": rpc error: code = NotFound desc = could not find container \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": container with ID starting with 46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22 not found: ID does not exist"
Jul 28 23:24:46.971888 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971792    2581 scope.go:117] "RemoveContainer" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.971998 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971971    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"} err="failed to get container status \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": rpc error: code = NotFound desc = could not find container \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": container with ID starting with 25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e not found: ID does not exist"
Jul 28 23:24:46.971998 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.971997    2581 scope.go:117] "RemoveContainer" containerID="2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"
Jul 28 23:24:46.972164 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 23:24:46.972244 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972221    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946"} err="failed to get container status \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": rpc error: code = NotFound desc = could not find container \"2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946\": container with ID starting with 2acb81d8ffcefaa1154884c5db69458591e9993e82691859a2db5e99b3643946 not found: ID does not exist"
Jul 28 23:24:46.972292 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972246    2581 scope.go:117] "RemoveContainer" containerID="e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"
Jul 28 23:24:46.972435 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 23:24:46.972554 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972535    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd"} err="failed to get container status \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": rpc error: code = NotFound desc = could not find container \"e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd\": container with ID starting with e954ee17e3a98730f7470b16b816b784ba39b280cfbe3390893404bc6175e4cd not found: ID does not exist"
Jul 28 23:24:46.972638 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972552    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 23:24:46.972638 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972567    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 23:24:46.972638 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972558    2581 scope.go:117] "RemoveContainer" containerID="d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"
Jul 28 23:24:46.972901 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972878    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1"} err="failed to get container status \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": rpc error: code = NotFound desc = could not find container \"d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1\": container with ID starting with d2ba69a48ddd3b5e7f2a7c9d801e254abb5fa8ec2076f938e4c948dec9428ab1 not found: ID does not exist"
Jul 28 23:24:46.972901 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.972901    2581 scope.go:117] "RemoveContainer" containerID="5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"
Jul 28 23:24:46.973178 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.973155    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477"} err="failed to get container status \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": rpc error: code = NotFound desc = could not find container \"5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477\": container with ID starting with 5a0d460b226b214285fc1656c707095146b01c7d0bcd48e9ce5b5b9da50df477 not found: ID does not exist"
Jul 28 23:24:46.973252 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.973181    2581 scope.go:117] "RemoveContainer" containerID="02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"
Jul 28 23:24:46.973461 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.973442    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c"} err="failed to get container status \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": rpc error: code = NotFound desc = could not find container \"02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c\": container with ID starting with 02d8362685ee10fd9858edd562276e563d45951dafa64de42083d9aa9f24a92c not found: ID does not exist"
Jul 28 23:24:46.973530 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.973461    2581 scope.go:117] "RemoveContainer" containerID="46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"
Jul 28 23:24:46.973801 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.973773    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22"} err="failed to get container status \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": rpc error: code = NotFound desc = could not find container \"46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22\": container with ID starting with 46232aa8edf52146302d95350c9d6dbfbb219e26acf644188aa47749f9199f22 not found: ID does not exist"
Jul 28 23:24:46.973873 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.973805    2581 scope.go:117] "RemoveContainer" containerID="25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"
Jul 28 23:24:46.974123 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.974104    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e"} err="failed to get container status \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": rpc error: code = NotFound desc = could not find container \"25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e\": container with ID starting with 25f028a984bfed1228906b45b1679508a65b6c693fd75af739e9841784d5fd7e not found: ID does not exist"
Jul 28 23:24:46.974830 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.974811    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 23:24:46.976214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.976197    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 23:24:46.981122 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.981098    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25" path="/var/lib/kubelet/pods/4fd0ce57-e07d-4017-8e8b-b1daa9bf1d25/volumes"
Jul 28 23:24:46.981663 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.981645    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:24:46.995448 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6540b711-08f7-48cc-9f7a-668d98d894e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995529 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995529 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6540b711-08f7-48cc-9f7a-668d98d894e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995529 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-config\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995653 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995772 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995772 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rzgc\" (UniqueName: \"kubernetes.io/projected/6540b711-08f7-48cc-9f7a-668d98d894e6-kube-api-access-7rzgc\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995772 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995772 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995772 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995926 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995926 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995926 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995926 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.995926 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.996093 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.996093 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.995998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:46.996093 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:46.996039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097374 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6540b711-08f7-48cc-9f7a-668d98d894e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097374 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-config\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rzgc\" (UniqueName: \"kubernetes.io/projected/6540b711-08f7-48cc-9f7a-668d98d894e6-kube-api-access-7rzgc\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.097605 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6540b711-08f7-48cc-9f7a-668d98d894e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.097975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098030 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.098021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.098558 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.098198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.099106 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.098757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.099106 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.098802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.099427 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.099399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.100121 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.100093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.100304 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.100274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.101427 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.100994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.101427 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.101389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.101427 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.101410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-config\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.101645 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.101468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6540b711-08f7-48cc-9f7a-668d98d894e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.101645 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.101482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.101979 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.101944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6540b711-08f7-48cc-9f7a-668d98d894e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.102272 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.102248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.102733 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.102703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6540b711-08f7-48cc-9f7a-668d98d894e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.103238 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.103215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.103386 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.103366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.103543 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.103526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.103638 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.103606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6540b711-08f7-48cc-9f7a-668d98d894e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.106187 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.106169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rzgc\" (UniqueName: \"kubernetes.io/projected/6540b711-08f7-48cc-9f7a-668d98d894e6-kube-api-access-7rzgc\") pod \"prometheus-k8s-0\" (UID: \"6540b711-08f7-48cc-9f7a-668d98d894e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.278885 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.278800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:24:47.416583 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.416404    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 23:24:47.418861 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:24:47.418834    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6540b711_08f7_48cc_9f7a_668d98d894e6.slice/crio-78ca9a9453486331f1866b51606c9d62e27477dcfc7d9c958ae3119224ffc336 WatchSource:0}: Error finding container 78ca9a9453486331f1866b51606c9d62e27477dcfc7d9c958ae3119224ffc336: Status 404 returned error can't find the container with id 78ca9a9453486331f1866b51606c9d62e27477dcfc7d9c958ae3119224ffc336
Jul 28 23:24:47.917876 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.917840    2581 generic.go:356] "Generic (PLEG): container finished" podID="6540b711-08f7-48cc-9f7a-668d98d894e6" containerID="7728f922f63832fe347860904c9233ea5babb15a83f84eab58a4976d235d3a17" exitCode=0
Jul 28 23:24:47.918063 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.917919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerDied","Data":"7728f922f63832fe347860904c9233ea5babb15a83f84eab58a4976d235d3a17"}
Jul 28 23:24:47.918063 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:47.917987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerStarted","Data":"78ca9a9453486331f1866b51606c9d62e27477dcfc7d9c958ae3119224ffc336"}
Jul 28 23:24:48.927031 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:48.926990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerStarted","Data":"bf9ab2b5a607c03d98ffa8ea7deba65db55bcfc14e98cc91013f9391acdbbc98"}
Jul 28 23:24:48.927493 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:48.927113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerStarted","Data":"36ed18fd457e60599b08c8ec5bdc02eedf20857195fc58de21b653178604884f"}
Jul 28 23:24:48.927493 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:48.927144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerStarted","Data":"3f781e605c34132a3bd949c6facdaee3c067f0cae7c589c0e6216db015f05673"}
Jul 28 23:24:48.927493 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:48.927178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerStarted","Data":"f045ed053a71b0ded17dddda30dd3b239ddb355265324e1eb36d5393dd2fcce9"}
Jul 28 23:24:48.927493 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:48.927209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerStarted","Data":"8358e4d144150652ae0a452c095437803d542244c23a13302a80b20e584c6664"}
Jul 28 23:24:48.927493 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:48.927233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6540b711-08f7-48cc-9f7a-668d98d894e6","Type":"ContainerStarted","Data":"21de8b8938fcbf6c66b0dc35952ddef62c358643ed7fd7236ae0e7383e27f279"}
Jul 28 23:24:48.954216 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:48.954159    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9360332380000003 podStartE2EDuration="2.954140651s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.107413ms" isStatefulPod=false podCreationTimestamp="2026-07-28 23:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:24:48.95136026 +0000 UTC m=+184.453906791" watchObservedRunningTime="2026-07-28 23:24:48.954140651 +0000 UTC m=+184.456687207"
Jul 28 23:24:52.279987 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:24:52.279929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:25:47.279778 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:25:47.279740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:25:47.295047 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:25:47.295022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:25:48.176297 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:25:48.176263    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 23:26:44.850112 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:26:44.850082    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 23:27:23.504699 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.504596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"]
Jul 28 23:27:23.507823 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.507807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.510536 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.510512    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 23:27:23.510671 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.510516    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 23:27:23.511397 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.511379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 23:27:23.511503 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.511483    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 23:27:23.511571 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.511503    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 23:27:23.511571 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.511562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gtszb\""
Jul 28 23:27:23.515600 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.515580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"]
Jul 28 23:27:23.522298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.522275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/645e35bd-5b03-4ccb-b252-71d9b6e01cee-cert\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.522392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.522312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj79p\" (UniqueName: \"kubernetes.io/projected/645e35bd-5b03-4ccb-b252-71d9b6e01cee-kube-api-access-zj79p\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.522392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.522357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/645e35bd-5b03-4ccb-b252-71d9b6e01cee-metrics-cert\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.522464 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.522397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/645e35bd-5b03-4ccb-b252-71d9b6e01cee-manager-config\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.623002 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.622965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/645e35bd-5b03-4ccb-b252-71d9b6e01cee-cert\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.623205 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.623013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj79p\" (UniqueName: \"kubernetes.io/projected/645e35bd-5b03-4ccb-b252-71d9b6e01cee-kube-api-access-zj79p\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.623205 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.623069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/645e35bd-5b03-4ccb-b252-71d9b6e01cee-metrics-cert\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.623205 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.623104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/645e35bd-5b03-4ccb-b252-71d9b6e01cee-manager-config\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.623829 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.623809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/645e35bd-5b03-4ccb-b252-71d9b6e01cee-manager-config\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.625518 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.625496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/645e35bd-5b03-4ccb-b252-71d9b6e01cee-metrics-cert\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.625667 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.625648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/645e35bd-5b03-4ccb-b252-71d9b6e01cee-cert\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.630814 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.630790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj79p\" (UniqueName: \"kubernetes.io/projected/645e35bd-5b03-4ccb-b252-71d9b6e01cee-kube-api-access-zj79p\") pod \"lws-controller-manager-7858699d7d-qjxws\" (UID: \"645e35bd-5b03-4ccb-b252-71d9b6e01cee\") " pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.817256 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.817156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:23.942758 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.942740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"]
Jul 28 23:27:23.945461 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:27:23.945429    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645e35bd_5b03_4ccb_b252_71d9b6e01cee.slice/crio-d58a3e80f945a040dd356bc90c0f026eedec08d3de9f1ead25dffb72e036f71c WatchSource:0}: Error finding container d58a3e80f945a040dd356bc90c0f026eedec08d3de9f1ead25dffb72e036f71c: Status 404 returned error can't find the container with id d58a3e80f945a040dd356bc90c0f026eedec08d3de9f1ead25dffb72e036f71c
Jul 28 23:27:23.947353 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:23.947336    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 23:27:24.531686 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:24.531650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws" event={"ID":"645e35bd-5b03-4ccb-b252-71d9b6e01cee","Type":"ContainerStarted","Data":"d58a3e80f945a040dd356bc90c0f026eedec08d3de9f1ead25dffb72e036f71c"}
Jul 28 23:27:27.546607 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:27.546567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws" event={"ID":"645e35bd-5b03-4ccb-b252-71d9b6e01cee","Type":"ContainerStarted","Data":"0d601aeaf575f9b2f4d85f5bf80f4f87f80215c16ddc639a2cc1650671d2f4ec"}
Jul 28 23:27:27.547079 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:27.546669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:27.562301 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:27.562249    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws" podStartSLOduration=1.765620749 podStartE2EDuration="4.562232813s" totalImagesPullingTime="2.796612064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:27:27.561360623 +0000 UTC m=+343.063907167" watchObservedRunningTime="2026-07-28 23:27:27.562232813 +0000 UTC m=+343.064779346"
Jul 28 23:27:32.180109 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.180071    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"]
Jul 28 23:27:32.183619 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.183593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.186090 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.186071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 23:27:32.186589 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.186569    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 23:27:32.186720 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.186670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7fb7f\""
Jul 28 23:27:32.186720 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.186705    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 23:27:32.186841 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.186775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 23:27:32.200970 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.200946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"]
Jul 28 23:27:32.298290 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.298258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.298290 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.298294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-tmp\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.298546 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.298322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.298546 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.298401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-manifests\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.298546 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.298440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m27tz\" (UniqueName: \"kubernetes.io/projected/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-kube-api-access-m27tz\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.399333 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.399294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.399333 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.399335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-tmp\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.399586 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.399375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.399586 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.399402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-manifests\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.399586 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.399430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m27tz\" (UniqueName: \"kubernetes.io/projected/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-kube-api-access-m27tz\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.399826 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.399803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-manifests\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.401793 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.401766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-tmp\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.401994 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.401977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-webhook-cert\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.402036 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.402010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.407258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.407234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m27tz\" (UniqueName: \"kubernetes.io/projected/dcd05d2a-9932-4c89-b1d3-68c0f4f0115d-kube-api-access-m27tz\") pod \"opendatahub-operator-controller-manager-dc9c5fb48-7hw5m\" (UID: \"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d\") " pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.493548 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.493480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:32.624515 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:32.624483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"]
Jul 28 23:27:32.627821 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:27:32.627795    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd05d2a_9932_4c89_b1d3_68c0f4f0115d.slice/crio-8281358150a3ab79b354417e87661f6c12f671d84de80dae40acf27783c62f83 WatchSource:0}: Error finding container 8281358150a3ab79b354417e87661f6c12f671d84de80dae40acf27783c62f83: Status 404 returned error can't find the container with id 8281358150a3ab79b354417e87661f6c12f671d84de80dae40acf27783c62f83
Jul 28 23:27:33.572092 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:33.572059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m" event={"ID":"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d","Type":"ContainerStarted","Data":"8281358150a3ab79b354417e87661f6c12f671d84de80dae40acf27783c62f83"}
Jul 28 23:27:36.593318 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:36.593278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m" event={"ID":"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d","Type":"ContainerStarted","Data":"d97ecee1f110956ae3e869f1d04269ce821e9fe6fe489f1a9b328c9fb9008339"}
Jul 28 23:27:37.599095 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:37.598995    2581 generic.go:356] "Generic (PLEG): container finished" podID="dcd05d2a-9932-4c89-b1d3-68c0f4f0115d" containerID="d97ecee1f110956ae3e869f1d04269ce821e9fe6fe489f1a9b328c9fb9008339" exitCode=0
Jul 28 23:27:37.599095 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:37.599078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m" event={"ID":"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d","Type":"ContainerDied","Data":"d97ecee1f110956ae3e869f1d04269ce821e9fe6fe489f1a9b328c9fb9008339"}
Jul 28 23:27:38.552531 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:38.552497    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7858699d7d-qjxws"
Jul 28 23:27:38.605204 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:38.605170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m" event={"ID":"dcd05d2a-9932-4c89-b1d3-68c0f4f0115d","Type":"ContainerStarted","Data":"6b5f2f6070fb835bfcbb0a6d599b7ee77c2932a71737259f8e6629924f58aeb5"}
Jul 28 23:27:38.605647 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:38.605300    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:38.623556 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:38.623507    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m" podStartSLOduration=2.171784433 podStartE2EDuration="6.623489994s" totalImagesPullingTime="3.388738134s" totalInitContainerRuntime="1.062967427s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:27:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:27:38.622180124 +0000 UTC m=+354.124726667" watchObservedRunningTime="2026-07-28 23:27:38.623489994 +0000 UTC m=+354.126036584"
Jul 28 23:27:49.613666 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:49.613615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dc9c5fb48-7hw5m"
Jul 28 23:27:51.824828 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.824796    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"]
Jul 28 23:27:51.835613 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.835581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:51.838208 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.838185    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 23:27:51.838378 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.838319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 23:27:51.838493 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.838477    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 23:27:51.838576 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.838555    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 23:27:51.838810 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.838785    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"]
Jul 28 23:27:51.839444 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.839424    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z8jkr\""
Jul 28 23:27:51.967711 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.967677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tmp\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:51.967872 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.967740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7zp\" (UniqueName: \"kubernetes.io/projected/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-kube-api-access-vb7zp\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:51.967915 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:51.967876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tls-certs\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.068558 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.068523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tmp\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.068771 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.068571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7zp\" (UniqueName: \"kubernetes.io/projected/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-kube-api-access-vb7zp\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.068771 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.068652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tls-certs\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.068889 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:27:52.068856    2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 23:27:52.068954 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:27:52.068947    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tls-certs podName:4c8fb409-49d6-453f-9c3e-e34db8aa6eb2 nodeName:}" failed. No retries permitted until 2026-07-28 23:27:52.568923357 +0000 UTC m=+368.071469883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tls-certs") pod "kube-auth-proxy-77f8c77c69-896nb" (UID: "4c8fb409-49d6-453f-9c3e-e34db8aa6eb2") : secret "kube-auth-proxy-tls" not found
Jul 28 23:27:52.070836 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.070813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tmp\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.079938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.079883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7zp\" (UniqueName: \"kubernetes.io/projected/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-kube-api-access-vb7zp\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.572854 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.572822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tls-certs\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.575314 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.575284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c8fb409-49d6-453f-9c3e-e34db8aa6eb2-tls-certs\") pod \"kube-auth-proxy-77f8c77c69-896nb\" (UID: \"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2\") " pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.746392 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.746353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"
Jul 28 23:27:52.892643 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:52.892599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77f8c77c69-896nb"]
Jul 28 23:27:52.894587 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:27:52.894555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8fb409_49d6_453f_9c3e_e34db8aa6eb2.slice/crio-cf8a49d4077c28370e9a720e1b82ad32e26b5ca8e91117a82e99509ccff6a032 WatchSource:0}: Error finding container cf8a49d4077c28370e9a720e1b82ad32e26b5ca8e91117a82e99509ccff6a032: Status 404 returned error can't find the container with id cf8a49d4077c28370e9a720e1b82ad32e26b5ca8e91117a82e99509ccff6a032
Jul 28 23:27:53.675453 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:53.675415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb" event={"ID":"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2","Type":"ContainerStarted","Data":"cf8a49d4077c28370e9a720e1b82ad32e26b5ca8e91117a82e99509ccff6a032"}
Jul 28 23:27:55.685309 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:55.685220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb" event={"ID":"4c8fb409-49d6-453f-9c3e-e34db8aa6eb2","Type":"ContainerStarted","Data":"c0f90bb0e944da51b93a41c88c1eaae31d25361a31359c549838d089d279d371"}
Jul 28 23:27:55.700444 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:27:55.700395    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77f8c77c69-896nb" podStartSLOduration=2.224748651 podStartE2EDuration="4.700379188s" totalImagesPullingTime="2.475630537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:27:55.699066337 +0000 UTC m=+371.201612849" watchObservedRunningTime="2026-07-28 23:27:55.700379188 +0000 UTC m=+371.202925725"
Jul 28 23:29:36.071904 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.071799    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"]
Jul 28 23:29:36.075278 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.075262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"
Jul 28 23:29:36.077725 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.077699    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8kh54\""
Jul 28 23:29:36.077725 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.077715    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 23:29:36.077894 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.077754    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 23:29:36.078794 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.078780    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 23:29:36.085612 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.085587    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"]
Jul 28 23:29:36.085730 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.085702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzg9\" (UniqueName: \"kubernetes.io/projected/ace06479-a5a9-480e-b560-c6875c2039ed-kube-api-access-vhzg9\") pod \"dns-operator-controller-manager-864dd78c4c-zlvzq\" (UID: \"ace06479-a5a9-480e-b560-c6875c2039ed\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"
Jul 28 23:29:36.186806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.186772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzg9\" (UniqueName: \"kubernetes.io/projected/ace06479-a5a9-480e-b560-c6875c2039ed-kube-api-access-vhzg9\") pod \"dns-operator-controller-manager-864dd78c4c-zlvzq\" (UID: \"ace06479-a5a9-480e-b560-c6875c2039ed\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"
Jul 28 23:29:36.195669 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.195621    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzg9\" (UniqueName: \"kubernetes.io/projected/ace06479-a5a9-480e-b560-c6875c2039ed-kube-api-access-vhzg9\") pod \"dns-operator-controller-manager-864dd78c4c-zlvzq\" (UID: \"ace06479-a5a9-480e-b560-c6875c2039ed\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"
Jul 28 23:29:36.385061 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.385029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"
Jul 28 23:29:36.509615 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:36.509592    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"]
Jul 28 23:29:36.511370 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:29:36.511335    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace06479_a5a9_480e_b560_c6875c2039ed.slice/crio-04c597747117cfa53889610c5590ef6e99c5b85160edf58366aff2819b051a63 WatchSource:0}: Error finding container 04c597747117cfa53889610c5590ef6e99c5b85160edf58366aff2819b051a63: Status 404 returned error can't find the container with id 04c597747117cfa53889610c5590ef6e99c5b85160edf58366aff2819b051a63
Jul 28 23:29:37.120474 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.120438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq" event={"ID":"ace06479-a5a9-480e-b560-c6875c2039ed","Type":"ContainerStarted","Data":"04c597747117cfa53889610c5590ef6e99c5b85160edf58366aff2819b051a63"}
Jul 28 23:29:37.554434 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.554363    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"]
Jul 28 23:29:37.557789 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.557769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.560303 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.560277    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6cltk\""
Jul 28 23:29:37.560303 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.560286    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 23:29:37.560477 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.560312    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 23:29:37.566845 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.566824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"]
Jul 28 23:29:37.598451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.598424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63112f2b-fcdd-41e2-9de1-151def88d354-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.598610 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.598491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qw8s\" (UniqueName: \"kubernetes.io/projected/63112f2b-fcdd-41e2-9de1-151def88d354-kube-api-access-4qw8s\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.598610 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.598539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63112f2b-fcdd-41e2-9de1-151def88d354-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.699445 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.699411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qw8s\" (UniqueName: \"kubernetes.io/projected/63112f2b-fcdd-41e2-9de1-151def88d354-kube-api-access-4qw8s\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.699642 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.699474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63112f2b-fcdd-41e2-9de1-151def88d354-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.699642 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.699508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63112f2b-fcdd-41e2-9de1-151def88d354-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.699770 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:29:37.699645    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 23:29:37.699770 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:29:37.699722    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63112f2b-fcdd-41e2-9de1-151def88d354-plugin-serving-cert podName:63112f2b-fcdd-41e2-9de1-151def88d354 nodeName:}" failed. No retries permitted until 2026-07-28 23:29:38.199700983 +0000 UTC m=+473.702247505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/63112f2b-fcdd-41e2-9de1-151def88d354-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-dbgc5" (UID: "63112f2b-fcdd-41e2-9de1-151def88d354") : secret "plugin-serving-cert" not found
Jul 28 23:29:37.702390 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.700341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63112f2b-fcdd-41e2-9de1-151def88d354-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:37.707316 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:37.707290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qw8s\" (UniqueName: \"kubernetes.io/projected/63112f2b-fcdd-41e2-9de1-151def88d354-kube-api-access-4qw8s\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:38.203782 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:38.203751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63112f2b-fcdd-41e2-9de1-151def88d354-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:38.205988 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:38.205965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63112f2b-fcdd-41e2-9de1-151def88d354-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dbgc5\" (UID: \"63112f2b-fcdd-41e2-9de1-151def88d354\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:38.478180 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:38.478091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"
Jul 28 23:29:38.714875 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:38.713020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5"]
Jul 28 23:29:39.130804 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:39.130761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5" event={"ID":"63112f2b-fcdd-41e2-9de1-151def88d354","Type":"ContainerStarted","Data":"fc8f3d0c459769f445f78b93a26033e93d81ecb9d945f2aa87d798a42973c82e"}
Jul 28 23:29:39.132167 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:39.132142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq" event={"ID":"ace06479-a5a9-480e-b560-c6875c2039ed","Type":"ContainerStarted","Data":"fe57e3e149f6d99dc62a7064dc5e42175425aa36dbd921bd80b4f09504b2850d"}
Jul 28 23:29:39.132294 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:39.132279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"
Jul 28 23:29:39.148178 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:39.148133    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq" podStartSLOduration=1.021701196 podStartE2EDuration="3.148119774s" totalImagesPullingTime="2.126418578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:29:39.146748594 +0000 UTC m=+474.649295125" watchObservedRunningTime="2026-07-28 23:29:39.148119774 +0000 UTC m=+474.650666305"
Jul 28 23:29:50.139733 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:29:50.139694    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zlvzq"
Jul 28 23:30:01.248004 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:01.247961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5" event={"ID":"63112f2b-fcdd-41e2-9de1-151def88d354","Type":"ContainerStarted","Data":"8138a6f9c13c2fc627354ac4e1b3dbf4f054ad696cae1848f0e6a3458c303033"}
Jul 28 23:30:01.264728 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:01.264679    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dbgc5" podStartSLOduration=2.23765414 podStartE2EDuration="24.2646635s" totalImagesPullingTime="22.02700936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:30:01.263421591 +0000 UTC m=+496.765968119" watchObservedRunningTime="2026-07-28 23:30:01.2646635 +0000 UTC m=+496.767210031"
Jul 28 23:30:22.225054 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.225023    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:30:22.227982 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.227966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.230761 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.230738    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 23:30:22.237887 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.237867    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:30:22.251221 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.251193    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:30:22.301443 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.301414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80689370-2339-4850-aa95-eac9cff6c354-config-file\") pod \"limitador-limitador-54d6d988d5-ctkwd\" (UID: \"80689370-2339-4850-aa95-eac9cff6c354\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.301604 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.301525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgs4g\" (UniqueName: \"kubernetes.io/projected/80689370-2339-4850-aa95-eac9cff6c354-kube-api-access-hgs4g\") pod \"limitador-limitador-54d6d988d5-ctkwd\" (UID: \"80689370-2339-4850-aa95-eac9cff6c354\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.402062 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.402033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgs4g\" (UniqueName: \"kubernetes.io/projected/80689370-2339-4850-aa95-eac9cff6c354-kube-api-access-hgs4g\") pod \"limitador-limitador-54d6d988d5-ctkwd\" (UID: \"80689370-2339-4850-aa95-eac9cff6c354\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.402188 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.402088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80689370-2339-4850-aa95-eac9cff6c354-config-file\") pod \"limitador-limitador-54d6d988d5-ctkwd\" (UID: \"80689370-2339-4850-aa95-eac9cff6c354\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.402683 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.402663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80689370-2339-4850-aa95-eac9cff6c354-config-file\") pod \"limitador-limitador-54d6d988d5-ctkwd\" (UID: \"80689370-2339-4850-aa95-eac9cff6c354\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.409805 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.409788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgs4g\" (UniqueName: \"kubernetes.io/projected/80689370-2339-4850-aa95-eac9cff6c354-kube-api-access-hgs4g\") pod \"limitador-limitador-54d6d988d5-ctkwd\" (UID: \"80689370-2339-4850-aa95-eac9cff6c354\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.538492 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.538415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:22.683219 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.683138    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:30:22.686032 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:30:22.685993    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80689370_2339_4850_aa95_eac9cff6c354.slice/crio-fb318b60305c0918cf319c9851ccd8b40019ff0bb5fd84e470b656f612cd2da5 WatchSource:0}: Error finding container fb318b60305c0918cf319c9851ccd8b40019ff0bb5fd84e470b656f612cd2da5: Status 404 returned error can't find the container with id fb318b60305c0918cf319c9851ccd8b40019ff0bb5fd84e470b656f612cd2da5
Jul 28 23:30:22.834383 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.834299    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-cfwd5"]
Jul 28 23:30:22.845779 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.845751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cfwd5"]
Jul 28 23:30:22.845923 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.845844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cfwd5"
Jul 28 23:30:22.848297 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.848268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jsx9f\""
Jul 28 23:30:22.906259 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:22.906228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdqc\" (UniqueName: \"kubernetes.io/projected/9cd64335-f98f-445c-8c81-f6f98e245d28-kube-api-access-jqdqc\") pod \"authorino-75d556bd77-cfwd5\" (UID: \"9cd64335-f98f-445c-8c81-f6f98e245d28\") " pod="kuadrant-system/authorino-75d556bd77-cfwd5"
Jul 28 23:30:23.006696 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:23.006652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdqc\" (UniqueName: \"kubernetes.io/projected/9cd64335-f98f-445c-8c81-f6f98e245d28-kube-api-access-jqdqc\") pod \"authorino-75d556bd77-cfwd5\" (UID: \"9cd64335-f98f-445c-8c81-f6f98e245d28\") " pod="kuadrant-system/authorino-75d556bd77-cfwd5"
Jul 28 23:30:23.014154 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:23.014121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdqc\" (UniqueName: \"kubernetes.io/projected/9cd64335-f98f-445c-8c81-f6f98e245d28-kube-api-access-jqdqc\") pod \"authorino-75d556bd77-cfwd5\" (UID: \"9cd64335-f98f-445c-8c81-f6f98e245d28\") " pod="kuadrant-system/authorino-75d556bd77-cfwd5"
Jul 28 23:30:23.156406 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:23.156376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cfwd5"
Jul 28 23:30:23.332813 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:23.332673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cfwd5"]
Jul 28 23:30:23.337859 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:30:23.337820    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd64335_f98f_445c_8c81_f6f98e245d28.slice/crio-df83ecca78bbd1dc4b06be5ce323abc633218c1fbfad599b71abbac0d3a3b2d6 WatchSource:0}: Error finding container df83ecca78bbd1dc4b06be5ce323abc633218c1fbfad599b71abbac0d3a3b2d6: Status 404 returned error can't find the container with id df83ecca78bbd1dc4b06be5ce323abc633218c1fbfad599b71abbac0d3a3b2d6
Jul 28 23:30:23.346812 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:23.346550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cfwd5" event={"ID":"9cd64335-f98f-445c-8c81-f6f98e245d28","Type":"ContainerStarted","Data":"df83ecca78bbd1dc4b06be5ce323abc633218c1fbfad599b71abbac0d3a3b2d6"}
Jul 28 23:30:23.348458 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:23.348410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd" event={"ID":"80689370-2339-4850-aa95-eac9cff6c354","Type":"ContainerStarted","Data":"fb318b60305c0918cf319c9851ccd8b40019ff0bb5fd84e470b656f612cd2da5"}
Jul 28 23:30:25.358853 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:25.358816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd" event={"ID":"80689370-2339-4850-aa95-eac9cff6c354","Type":"ContainerStarted","Data":"cd9cb2f4115f4a1c455d9812d5c2736d6eaf7434717ab7840e96e48d65c87d73"}
Jul 28 23:30:25.359227 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:25.358899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:25.374498 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:25.374455    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd" podStartSLOduration=0.815736074 podStartE2EDuration="3.374441698s" totalImagesPullingTime="2.558705624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:30:25.372827494 +0000 UTC m=+520.875374037" watchObservedRunningTime="2026-07-28 23:30:25.374441698 +0000 UTC m=+520.876988228"
Jul 28 23:30:31.388233 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:31.388196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cfwd5" event={"ID":"9cd64335-f98f-445c-8c81-f6f98e245d28","Type":"ContainerStarted","Data":"b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1"}
Jul 28 23:30:31.406656 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:31.406587    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-cfwd5" podStartSLOduration=2.406497325 podStartE2EDuration="9.40656998s" totalImagesPullingTime="7.000072655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:30:31.404963585 +0000 UTC m=+526.907510116" watchObservedRunningTime="2026-07-28 23:30:31.40656998 +0000 UTC m=+526.909116511"
Jul 28 23:30:36.364893 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:36.364861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"
Jul 28 23:30:51.512950 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.512872    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kh82g"]
Jul 28 23:30:51.516227 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.516209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kh82g"
Jul 28 23:30:51.522522 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.522495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kh82g"]
Jul 28 23:30:51.659792 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.659755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshrf\" (UniqueName: \"kubernetes.io/projected/f349974a-f7e3-46fb-a8b4-d1a59fc4c370-kube-api-access-vshrf\") pod \"authorino-5d4b67c446-kh82g\" (UID: \"f349974a-f7e3-46fb-a8b4-d1a59fc4c370\") " pod="kuadrant-system/authorino-5d4b67c446-kh82g"
Jul 28 23:30:51.685198 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.685162    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kh82g"]
Jul 28 23:30:51.685427 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:30:51.685408    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vshrf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-kh82g" podUID="f349974a-f7e3-46fb-a8b4-d1a59fc4c370"
Jul 28 23:30:51.709548 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.709520    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-kdqqz"]
Jul 28 23:30:51.712894 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.712879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:51.715341 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.715322    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 23:30:51.720056 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.720038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-kdqqz"]
Jul 28 23:30:51.761008 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.760981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshrf\" (UniqueName: \"kubernetes.io/projected/f349974a-f7e3-46fb-a8b4-d1a59fc4c370-kube-api-access-vshrf\") pod \"authorino-5d4b67c446-kh82g\" (UID: \"f349974a-f7e3-46fb-a8b4-d1a59fc4c370\") " pod="kuadrant-system/authorino-5d4b67c446-kh82g"
Jul 28 23:30:51.768131 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.768071    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-kdqqz"]
Jul 28 23:30:51.768326 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:30:51.768308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-v5548 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-v5548 tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-kdqqz" podUID="2f193c17-346c-40de-a367-3979e1ebead4"
Jul 28 23:30:51.771855 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.771832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshrf\" (UniqueName: \"kubernetes.io/projected/f349974a-f7e3-46fb-a8b4-d1a59fc4c370-kube-api-access-vshrf\") pod \"authorino-5d4b67c446-kh82g\" (UID: \"f349974a-f7e3-46fb-a8b4-d1a59fc4c370\") " pod="kuadrant-system/authorino-5d4b67c446-kh82g"
Jul 28 23:30:51.861400 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.861363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f193c17-346c-40de-a367-3979e1ebead4-tls-cert\") pod \"authorino-76877994-kdqqz\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") " pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:51.861560 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.861431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5548\" (UniqueName: \"kubernetes.io/projected/2f193c17-346c-40de-a367-3979e1ebead4-kube-api-access-v5548\") pod \"authorino-76877994-kdqqz\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") " pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:51.961871 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.961834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5548\" (UniqueName: \"kubernetes.io/projected/2f193c17-346c-40de-a367-3979e1ebead4-kube-api-access-v5548\") pod \"authorino-76877994-kdqqz\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") " pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:51.962060 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.961927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f193c17-346c-40de-a367-3979e1ebead4-tls-cert\") pod \"authorino-76877994-kdqqz\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") " pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:51.964376 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.964352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f193c17-346c-40de-a367-3979e1ebead4-tls-cert\") pod \"authorino-76877994-kdqqz\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") " pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:51.971035 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:51.970997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5548\" (UniqueName: \"kubernetes.io/projected/2f193c17-346c-40de-a367-3979e1ebead4-kube-api-access-v5548\") pod \"authorino-76877994-kdqqz\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") " pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:52.480812 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.480779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kh82g"
Jul 28 23:30:52.481002 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.480784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:52.487902 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.487877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kh82g"
Jul 28 23:30:52.491036 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.491014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:52.567937 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.567903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f193c17-346c-40de-a367-3979e1ebead4-tls-cert\") pod \"2f193c17-346c-40de-a367-3979e1ebead4\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") "
Jul 28 23:30:52.568400 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.567964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5548\" (UniqueName: \"kubernetes.io/projected/2f193c17-346c-40de-a367-3979e1ebead4-kube-api-access-v5548\") pod \"2f193c17-346c-40de-a367-3979e1ebead4\" (UID: \"2f193c17-346c-40de-a367-3979e1ebead4\") "
Jul 28 23:30:52.568400 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.568037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vshrf\" (UniqueName: \"kubernetes.io/projected/f349974a-f7e3-46fb-a8b4-d1a59fc4c370-kube-api-access-vshrf\") pod \"f349974a-f7e3-46fb-a8b4-d1a59fc4c370\" (UID: \"f349974a-f7e3-46fb-a8b4-d1a59fc4c370\") "
Jul 28 23:30:52.570093 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.570060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f193c17-346c-40de-a367-3979e1ebead4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2f193c17-346c-40de-a367-3979e1ebead4" (UID: "2f193c17-346c-40de-a367-3979e1ebead4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:30:52.570093 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.570075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f193c17-346c-40de-a367-3979e1ebead4-kube-api-access-v5548" (OuterVolumeSpecName: "kube-api-access-v5548") pod "2f193c17-346c-40de-a367-3979e1ebead4" (UID: "2f193c17-346c-40de-a367-3979e1ebead4"). InnerVolumeSpecName "kube-api-access-v5548". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:30:52.570238 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.570131    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f349974a-f7e3-46fb-a8b4-d1a59fc4c370-kube-api-access-vshrf" (OuterVolumeSpecName: "kube-api-access-vshrf") pod "f349974a-f7e3-46fb-a8b4-d1a59fc4c370" (UID: "f349974a-f7e3-46fb-a8b4-d1a59fc4c370"). InnerVolumeSpecName "kube-api-access-vshrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:30:52.668834 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.668803    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vshrf\" (UniqueName: \"kubernetes.io/projected/f349974a-f7e3-46fb-a8b4-d1a59fc4c370-kube-api-access-vshrf\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:30:52.668834 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.668829    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f193c17-346c-40de-a367-3979e1ebead4-tls-cert\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:30:52.668834 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:52.668840    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5548\" (UniqueName: \"kubernetes.io/projected/2f193c17-346c-40de-a367-3979e1ebead4-kube-api-access-v5548\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:30:53.029612 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.029578    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cfwd5"]
Jul 28 23:30:53.029862 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.029813    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-cfwd5" podUID="9cd64335-f98f-445c-8c81-f6f98e245d28" containerName="authorino" containerID="cri-o://b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1" gracePeriod=30
Jul 28 23:30:53.280097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.280067    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cfwd5"
Jul 28 23:30:53.375797 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.375764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqdqc\" (UniqueName: \"kubernetes.io/projected/9cd64335-f98f-445c-8c81-f6f98e245d28-kube-api-access-jqdqc\") pod \"9cd64335-f98f-445c-8c81-f6f98e245d28\" (UID: \"9cd64335-f98f-445c-8c81-f6f98e245d28\") "
Jul 28 23:30:53.377928 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.377892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd64335-f98f-445c-8c81-f6f98e245d28-kube-api-access-jqdqc" (OuterVolumeSpecName: "kube-api-access-jqdqc") pod "9cd64335-f98f-445c-8c81-f6f98e245d28" (UID: "9cd64335-f98f-445c-8c81-f6f98e245d28"). InnerVolumeSpecName "kube-api-access-jqdqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:30:53.476562 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.476530    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqdqc\" (UniqueName: \"kubernetes.io/projected/9cd64335-f98f-445c-8c81-f6f98e245d28-kube-api-access-jqdqc\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:30:53.486400 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.486376    2581 generic.go:356] "Generic (PLEG): container finished" podID="9cd64335-f98f-445c-8c81-f6f98e245d28" containerID="b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1" exitCode=0
Jul 28 23:30:53.486518 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.486423    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cfwd5"
Jul 28 23:30:53.486518 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.486464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cfwd5" event={"ID":"9cd64335-f98f-445c-8c81-f6f98e245d28","Type":"ContainerDied","Data":"b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1"}
Jul 28 23:30:53.486518 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.486506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cfwd5" event={"ID":"9cd64335-f98f-445c-8c81-f6f98e245d28","Type":"ContainerDied","Data":"df83ecca78bbd1dc4b06be5ce323abc633218c1fbfad599b71abbac0d3a3b2d6"}
Jul 28 23:30:53.486637 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.486525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-kdqqz"
Jul 28 23:30:53.486675 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.486525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kh82g"
Jul 28 23:30:53.486762 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.486528    2581 scope.go:117] "RemoveContainer" containerID="b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1"
Jul 28 23:30:53.496871 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.496850    2581 scope.go:117] "RemoveContainer" containerID="b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1"
Jul 28 23:30:53.497579 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:30:53.497546    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1\": container with ID starting with b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1 not found: ID does not exist" containerID="b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1"
Jul 28 23:30:53.497691 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.497600    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1"} err="failed to get container status \"b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1\": rpc error: code = NotFound desc = could not find container \"b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1\": container with ID starting with b55eeb10284ead2796c9bf6f5fc96f87b2b4be9df1b537fd119828ad74271cd1 not found: ID does not exist"
Jul 28 23:30:53.513943 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.513879    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kh82g"]
Jul 28 23:30:53.515322 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.515302    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kh82g"]
Jul 28 23:30:53.535093 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.535068    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-kdqqz"]
Jul 28 23:30:53.537976 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.537956    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-kdqqz"]
Jul 28 23:30:53.546879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.546860    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cfwd5"]
Jul 28 23:30:53.549046 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:53.549029    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cfwd5"]
Jul 28 23:30:54.981606 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:54.981559    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f193c17-346c-40de-a367-3979e1ebead4" path="/var/lib/kubelet/pods/2f193c17-346c-40de-a367-3979e1ebead4/volumes"
Jul 28 23:30:54.982097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:54.981931    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd64335-f98f-445c-8c81-f6f98e245d28" path="/var/lib/kubelet/pods/9cd64335-f98f-445c-8c81-f6f98e245d28/volumes"
Jul 28 23:30:54.982324 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:30:54.982305    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f349974a-f7e3-46fb-a8b4-d1a59fc4c370" path="/var/lib/kubelet/pods/f349974a-f7e3-46fb-a8b4-d1a59fc4c370/volumes"
Jul 28 23:31:34.763183 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:31:34.763149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:31:54.238460 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:31:54.238422    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:32:13.220349 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:13.220273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:32:26.098893 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.098863    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"]
Jul 28 23:32:26.100398 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.099212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd64335-f98f-445c-8c81-f6f98e245d28" containerName="authorino"
Jul 28 23:32:26.100398 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.099224    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd64335-f98f-445c-8c81-f6f98e245d28" containerName="authorino"
Jul 28 23:32:26.100398 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.099278    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd64335-f98f-445c-8c81-f6f98e245d28" containerName="authorino"
Jul 28 23:32:26.100886 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.100871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.104540 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.104514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 23:32:26.104691 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.104521    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 23:32:26.104691 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.104599    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dlh7r\""
Jul 28 23:32:26.104811 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.104643    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 23:32:26.113200 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.113181    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"]
Jul 28 23:32:26.192553 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.192523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.192716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.192575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.192716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.192666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9ztj\" (UniqueName: \"kubernetes.io/projected/e989e357-dbee-4c4b-9a35-2c2346d272a2-kube-api-access-w9ztj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.192716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.192696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.192874 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.192723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e989e357-dbee-4c4b-9a35-2c2346d272a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.192874 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.192754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293355 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293492 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9ztj\" (UniqueName: \"kubernetes.io/projected/e989e357-dbee-4c4b-9a35-2c2346d272a2-kube-api-access-w9ztj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293492 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293492 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e989e357-dbee-4c4b-9a35-2c2346d272a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293492 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293733 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293793 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293846 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.293897 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.293886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.295707 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.295682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e989e357-dbee-4c4b-9a35-2c2346d272a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.295978 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.295959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e989e357-dbee-4c4b-9a35-2c2346d272a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.300458 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.300440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9ztj\" (UniqueName: \"kubernetes.io/projected/e989e357-dbee-4c4b-9a35-2c2346d272a2-kube-api-access-w9ztj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9dmmv\" (UID: \"e989e357-dbee-4c4b-9a35-2c2346d272a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.411232 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.411204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:26.545039 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.545015    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"]
Jul 28 23:32:26.547229 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:32:26.547184    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode989e357_dbee_4c4b_9a35_2c2346d272a2.slice/crio-a319020f14d6bce023fc559d834c28b880dfcc9d4430a9450e60f012e2680bdb WatchSource:0}: Error finding container a319020f14d6bce023fc559d834c28b880dfcc9d4430a9450e60f012e2680bdb: Status 404 returned error can't find the container with id a319020f14d6bce023fc559d834c28b880dfcc9d4430a9450e60f012e2680bdb
Jul 28 23:32:26.548479 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.548451    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 23:32:26.896402 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:26.896317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv" event={"ID":"e989e357-dbee-4c4b-9a35-2c2346d272a2","Type":"ContainerStarted","Data":"a319020f14d6bce023fc559d834c28b880dfcc9d4430a9450e60f012e2680bdb"}
Jul 28 23:32:27.211188 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:27.211116    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:32:27.902102 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:27.902063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv" event={"ID":"e989e357-dbee-4c4b-9a35-2c2346d272a2","Type":"ContainerStarted","Data":"887cbf6ea48d935d512752aa71e59a6d0b5cfde2628d191eeed90c069cd14423"}
Jul 28 23:32:27.919478 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:27.919430    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv" podStartSLOduration=0.920404881 podStartE2EDuration="1.91941374s" totalImagesPullingTime="999.008859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:32:27.917852936 +0000 UTC m=+643.420399466" watchObservedRunningTime="2026-07-28 23:32:27.91941374 +0000 UTC m=+643.421960271"
Jul 28 23:32:36.411489 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:36.411453    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:36.411936 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:36.411520    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:36.423815 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:36.423786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:36.952361 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:36.952333    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9dmmv"
Jul 28 23:32:52.425583 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:32:52.425544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:33:14.514340 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:14.514304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:33:23.403518 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.403436    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"]
Jul 28 23:33:23.406963 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.406941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.409837 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.409818    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 23:33:23.419153 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.419131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"]
Jul 28 23:33:23.422604 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.422583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.422704 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.422621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.422754 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.422722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.422798 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.422786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.422834 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.422814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.422880 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.422842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpgn\" (UniqueName: \"kubernetes.io/projected/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-kube-api-access-2jpgn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.523696 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.523657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.523696 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.523693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.523929 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.523722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpgn\" (UniqueName: \"kubernetes.io/projected/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-kube-api-access-2jpgn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.523929 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.523778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.523929 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.523834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.523929 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.523886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.524162 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.524078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.524162 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.524133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.524245 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.524199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.526094 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.526068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.526277 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.526259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.530930 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.530911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpgn\" (UniqueName: \"kubernetes.io/projected/64c28241-4e7c-4edf-ba09-7e6db4fba9ba-kube-api-access-2jpgn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp\" (UID: \"64c28241-4e7c-4edf-ba09-7e6db4fba9ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.716245 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.716152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:23.848326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:23.848298    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"]
Jul 28 23:33:23.850229 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:33:23.850201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64c28241_4e7c_4edf_ba09_7e6db4fba9ba.slice/crio-6c1ff998e0ff22797225055b4771a7a7edc74f9fbac759590a2918dec6afeccf WatchSource:0}: Error finding container 6c1ff998e0ff22797225055b4771a7a7edc74f9fbac759590a2918dec6afeccf: Status 404 returned error can't find the container with id 6c1ff998e0ff22797225055b4771a7a7edc74f9fbac759590a2918dec6afeccf
Jul 28 23:33:24.147188 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:24.147148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp" event={"ID":"64c28241-4e7c-4edf-ba09-7e6db4fba9ba","Type":"ContainerStarted","Data":"6f99a031795ee05925aa66d18d9dd9f9f79b80f9427917e54e520b4fcd2c1e06"}
Jul 28 23:33:24.147351 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:24.147196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp" event={"ID":"64c28241-4e7c-4edf-ba09-7e6db4fba9ba","Type":"ContainerStarted","Data":"6c1ff998e0ff22797225055b4771a7a7edc74f9fbac759590a2918dec6afeccf"}
Jul 28 23:33:24.163930 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:24.163884    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp" podStartSLOduration=0.990485142 podStartE2EDuration="1.163869053s" totalImagesPullingTime="173.383911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:33:24.161870431 +0000 UTC m=+699.664416963" watchObservedRunningTime="2026-07-28 23:33:24.163869053 +0000 UTC m=+699.666415584"
Jul 28 23:33:26.413043 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:26.413005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:33:33.716822 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:33.716775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:33.717342 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:33.717033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:33.729173 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:33.729145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:33:34.201235 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:33:34.201205    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp"
Jul 28 23:34:55.516585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:34:55.516551    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:35:05.707050 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:35:05.707014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:35:16.622526 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:35:16.622487    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:36:58.312746 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:36:58.312707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:37:07.621215 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:37:07.621179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:37:37.111849 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:37:37.111811    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:37:46.512199 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:37:46.512155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:37:57.114657 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:37:57.114607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:38:04.809859 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:38:04.809824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:38:16.413830 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:38:16.413796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:38:26.221044 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:38:26.221007    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:38:37.220370 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:38:37.220336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:38:47.011335 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:38:47.011299    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:39:08.516554 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:39:08.516518    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:39:19.217496 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:39:19.217463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:39:42.636883 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:39:42.636793    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:39:56.211002 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:39:56.210967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:40:07.307048 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:40:07.307009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:40:25.415956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:40:25.415919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:40:37.414203 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:40:37.414168    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:40:47.416189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:40:47.416154    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:40:57.408411 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:40:57.408376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:41:08.711294 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:41:08.711210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:41:21.409417 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:41:21.409378    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:41:31.105000 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:41:31.104960    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:43:10.315802 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:43:10.315756    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:43:21.611571 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:43:21.611532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:43:37.513407 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:43:37.513373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:43:53.417937 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:43:53.417900    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:44:04.516552 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:44:04.516520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:44:14.014773 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:44:14.014736    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:44:25.606664 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:44:25.606614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:44:35.407424 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:44:35.407388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:44:44.115823 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:44:44.115786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:44:55.311718 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:44:55.311686    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:45:32.113140 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:32.113104    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:45:41.916160 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:41.916073    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:45:53.798821 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.798782    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"]
Jul 28 23:45:53.803927 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.803894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:53.806997 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.806968    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bfdf0414-kserve-self-signed-certs\""
Jul 28 23:45:53.811931 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.811903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"]
Jul 28 23:45:53.930760 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.930728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-dshm\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:53.930964 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.930784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-tmp-dir\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:53.930964 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.930812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7426ff66-6891-4ebe-9e08-a832111977f2-tls-certs\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:53.930964 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.930844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6984p\" (UniqueName: \"kubernetes.io/projected/7426ff66-6891-4ebe-9e08-a832111977f2-kube-api-access-6984p\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:53.930964 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.930883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-home\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:53.930964 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:53.930912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-model-cache\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.031727 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.031696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-dshm\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.031879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.031744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-tmp-dir\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.031879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.031761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7426ff66-6891-4ebe-9e08-a832111977f2-tls-certs\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.031879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.031786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6984p\" (UniqueName: \"kubernetes.io/projected/7426ff66-6891-4ebe-9e08-a832111977f2-kube-api-access-6984p\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.031879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.031818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-home\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.031879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.031849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-model-cache\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.032233 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.032209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-tmp-dir\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.032312 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.032287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-home\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.032361 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.032317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-model-cache\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.034161 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.034134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-dshm\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.034284 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.034269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7426ff66-6891-4ebe-9e08-a832111977f2-tls-certs\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.038981 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.038959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6984p\" (UniqueName: \"kubernetes.io/projected/7426ff66-6891-4ebe-9e08-a832111977f2-kube-api-access-6984p\") pod \"e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") " pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.115621 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.115589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:45:54.241824 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.241799    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"]
Jul 28 23:45:54.244261 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:45:54.244228    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7426ff66_6891_4ebe_9e08_a832111977f2.slice/crio-fbe06ba8267ca2f46df8c8d5b0fab3d7ae1f5173e0120f2ab1da855880d380bb WatchSource:0}: Error finding container fbe06ba8267ca2f46df8c8d5b0fab3d7ae1f5173e0120f2ab1da855880d380bb: Status 404 returned error can't find the container with id fbe06ba8267ca2f46df8c8d5b0fab3d7ae1f5173e0120f2ab1da855880d380bb
Jul 28 23:45:54.540920 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.540829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9" event={"ID":"7426ff66-6891-4ebe-9e08-a832111977f2","Type":"ContainerStarted","Data":"730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d"}
Jul 28 23:45:54.540920 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.540876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9" event={"ID":"7426ff66-6891-4ebe-9e08-a832111977f2","Type":"ContainerStarted","Data":"fbe06ba8267ca2f46df8c8d5b0fab3d7ae1f5173e0120f2ab1da855880d380bb"}
Jul 28 23:45:54.556487 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:54.556440    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9" podStartSLOduration=1.556395865 podStartE2EDuration="1.556395865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:45:54.555267366 +0000 UTC m=+1450.057813897" watchObservedRunningTime="2026-07-28 23:45:54.556395865 +0000 UTC m=+1450.058942395"
Jul 28 23:45:57.112558 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:45:57.112522    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:46:04.115911 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:04.115868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:46:04.116449 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:04.115926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:46:04.128486 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:04.128453    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:46:04.599479 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:04.599391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:46:20.811756 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.811722    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"]
Jul 28 23:46:20.815369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.815346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.817892 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.817869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bfdf0414-kserve-self-signed-certs\""
Jul 28 23:46:20.824844 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.824820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"]
Jul 28 23:46:20.873475 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.873437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-home\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.873716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.873497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-model-cache\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.873716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.873538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-dshm\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.873716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.873565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwc5\" (UniqueName: \"kubernetes.io/projected/cf3f657a-f9d1-4579-b5ca-85813d334d85-kube-api-access-4nwc5\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.873716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.873604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3f657a-f9d1-4579-b5ca-85813d334d85-tls-certs\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.873716 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.873706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-tmp-dir\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.974330 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.974286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-model-cache\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.974330 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.974337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-dshm\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.974585 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.974511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwc5\" (UniqueName: \"kubernetes.io/projected/cf3f657a-f9d1-4579-b5ca-85813d334d85-kube-api-access-4nwc5\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.974681 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.974582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3f657a-f9d1-4579-b5ca-85813d334d85-tls-certs\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.974744 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.974696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-model-cache\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.974744 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.974711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-tmp-dir\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.974838 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.974783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-home\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.975060 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.975033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-tmp-dir\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.975060 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.975059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-home\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.976573 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.976551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-dshm\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.977062 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.977042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3f657a-f9d1-4579-b5ca-85813d334d85-tls-certs\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:20.982611 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:20.982582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwc5\" (UniqueName: \"kubernetes.io/projected/cf3f657a-f9d1-4579-b5ca-85813d334d85-kube-api-access-4nwc5\") pod \"e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") " pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:21.126324 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:21.126291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:21.255781 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:21.255759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"]
Jul 28 23:46:21.257882 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:46:21.257855    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf3f657a_f9d1_4579_b5ca_85813d334d85.slice/crio-44bebe9964a4a99f69f78392e3e02abb71ac5b7249391f31a2681251d2ce34f3 WatchSource:0}: Error finding container 44bebe9964a4a99f69f78392e3e02abb71ac5b7249391f31a2681251d2ce34f3: Status 404 returned error can't find the container with id 44bebe9964a4a99f69f78392e3e02abb71ac5b7249391f31a2681251d2ce34f3
Jul 28 23:46:21.667858 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:21.667818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65" event={"ID":"cf3f657a-f9d1-4579-b5ca-85813d334d85","Type":"ContainerStarted","Data":"62317497fc1ac73f7845de4f008ed27364ce0eadf6c2b2c7fc4fc36b7fe9381e"}
Jul 28 23:46:21.668059 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:21.667866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65" event={"ID":"cf3f657a-f9d1-4579-b5ca-85813d334d85","Type":"ContainerStarted","Data":"44bebe9964a4a99f69f78392e3e02abb71ac5b7249391f31a2681251d2ce34f3"}
Jul 28 23:46:21.685303 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:21.685249    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65" podStartSLOduration=1.685231355 podStartE2EDuration="1.685231355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:46:21.68320016 +0000 UTC m=+1477.185746694" watchObservedRunningTime="2026-07-28 23:46:21.685231355 +0000 UTC m=+1477.187777886"
Jul 28 23:46:23.516156 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:23.516121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:46:31.126781 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:31.126751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:31.126781 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:31.126800    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:31.139541 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:31.139516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:46:31.724397 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:46:31.724369    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:47:09.175148 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:09.175062    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"]
Jul 28 23:47:10.709731 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:10.709700    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"]
Jul 28 23:47:13.320021 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:13.319986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:47:24.194295 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.194257    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9" podUID="7426ff66-6891-4ebe-9e08-a832111977f2" containerName="main" containerID="cri-o://730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d" gracePeriod=45
Jul 28 23:47:24.448949 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.448889    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:47:24.568022 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.567988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-dshm\") pod \"7426ff66-6891-4ebe-9e08-a832111977f2\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") "
Jul 28 23:47:24.568188 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-home\") pod \"7426ff66-6891-4ebe-9e08-a832111977f2\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") "
Jul 28 23:47:24.568188 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6984p\" (UniqueName: \"kubernetes.io/projected/7426ff66-6891-4ebe-9e08-a832111977f2-kube-api-access-6984p\") pod \"7426ff66-6891-4ebe-9e08-a832111977f2\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") "
Jul 28 23:47:24.568188 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-tmp-dir\") pod \"7426ff66-6891-4ebe-9e08-a832111977f2\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") "
Jul 28 23:47:24.568188 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7426ff66-6891-4ebe-9e08-a832111977f2-tls-certs\") pod \"7426ff66-6891-4ebe-9e08-a832111977f2\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") "
Jul 28 23:47:24.568385 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-model-cache\") pod \"7426ff66-6891-4ebe-9e08-a832111977f2\" (UID: \"7426ff66-6891-4ebe-9e08-a832111977f2\") "
Jul 28 23:47:24.568434 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568392    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-home" (OuterVolumeSpecName: "home") pod "7426ff66-6891-4ebe-9e08-a832111977f2" (UID: "7426ff66-6891-4ebe-9e08-a832111977f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:24.568434 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7426ff66-6891-4ebe-9e08-a832111977f2" (UID: "7426ff66-6891-4ebe-9e08-a832111977f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:24.568551 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568512    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7426ff66-6891-4ebe-9e08-a832111977f2" (UID: "7426ff66-6891-4ebe-9e08-a832111977f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:24.568697 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568664    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:24.568697 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568684    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:24.568697 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.568697    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:24.570298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.570274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-dshm" (OuterVolumeSpecName: "dshm") pod "7426ff66-6891-4ebe-9e08-a832111977f2" (UID: "7426ff66-6891-4ebe-9e08-a832111977f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:24.570298 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.570290    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7426ff66-6891-4ebe-9e08-a832111977f2-kube-api-access-6984p" (OuterVolumeSpecName: "kube-api-access-6984p") pod "7426ff66-6891-4ebe-9e08-a832111977f2" (UID: "7426ff66-6891-4ebe-9e08-a832111977f2"). InnerVolumeSpecName "kube-api-access-6984p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:47:24.570442 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.570290    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7426ff66-6891-4ebe-9e08-a832111977f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7426ff66-6891-4ebe-9e08-a832111977f2" (UID: "7426ff66-6891-4ebe-9e08-a832111977f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:47:24.669750 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.669714    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7426ff66-6891-4ebe-9e08-a832111977f2-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:24.669750 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.669742    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6984p\" (UniqueName: \"kubernetes.io/projected/7426ff66-6891-4ebe-9e08-a832111977f2-kube-api-access-6984p\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:24.669750 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.669754    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7426ff66-6891-4ebe-9e08-a832111977f2-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:24.956077 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.956039    2581 generic.go:356] "Generic (PLEG): container finished" podID="7426ff66-6891-4ebe-9e08-a832111977f2" containerID="730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d" exitCode=0
Jul 28 23:47:24.956258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.956082    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"
Jul 28 23:47:24.956258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.956127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9" event={"ID":"7426ff66-6891-4ebe-9e08-a832111977f2","Type":"ContainerDied","Data":"730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d"}
Jul 28 23:47:24.956258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.956165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9" event={"ID":"7426ff66-6891-4ebe-9e08-a832111977f2","Type":"ContainerDied","Data":"fbe06ba8267ca2f46df8c8d5b0fab3d7ae1f5173e0120f2ab1da855880d380bb"}
Jul 28 23:47:24.956258 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.956184    2581 scope.go:117] "RemoveContainer" containerID="730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d"
Jul 28 23:47:24.964749 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.964731    2581 scope.go:117] "RemoveContainer" containerID="730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d"
Jul 28 23:47:24.964998 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:47:24.964976    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d\": container with ID starting with 730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d not found: ID does not exist" containerID="730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d"
Jul 28 23:47:24.965042 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.965010    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d"} err="failed to get container status \"730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d\": rpc error: code = NotFound desc = could not find container \"730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d\": container with ID starting with 730bba7ed1dbc69013106f3929c303c7b2c55101d541117d4e0af4454a16219d not found: ID does not exist"
Jul 28 23:47:24.986272 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.986247    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"]
Jul 28 23:47:24.986272 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:24.986277    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bfdf0414-kserve-846d9688f5-hg7s9"]
Jul 28 23:47:25.728336 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:25.728286    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65" podUID="cf3f657a-f9d1-4579-b5ca-85813d334d85" containerName="main" containerID="cri-o://62317497fc1ac73f7845de4f008ed27364ce0eadf6c2b2c7fc4fc36b7fe9381e" gracePeriod=45
Jul 28 23:47:25.963524 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:25.963501    2581 generic.go:356] "Generic (PLEG): container finished" podID="cf3f657a-f9d1-4579-b5ca-85813d334d85" containerID="62317497fc1ac73f7845de4f008ed27364ce0eadf6c2b2c7fc4fc36b7fe9381e" exitCode=0
Jul 28 23:47:25.963700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:25.963570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65" event={"ID":"cf3f657a-f9d1-4579-b5ca-85813d334d85","Type":"ContainerDied","Data":"62317497fc1ac73f7845de4f008ed27364ce0eadf6c2b2c7fc4fc36b7fe9381e"}
Jul 28 23:47:25.978220 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:25.978195    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:47:26.084785 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.084715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-model-cache\") pod \"cf3f657a-f9d1-4579-b5ca-85813d334d85\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") "
Jul 28 23:47:26.084785 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.084758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-home\") pod \"cf3f657a-f9d1-4579-b5ca-85813d334d85\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") "
Jul 28 23:47:26.084994 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.084806    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-dshm\") pod \"cf3f657a-f9d1-4579-b5ca-85813d334d85\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") "
Jul 28 23:47:26.084994 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.084863    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3f657a-f9d1-4579-b5ca-85813d334d85-tls-certs\") pod \"cf3f657a-f9d1-4579-b5ca-85813d334d85\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") "
Jul 28 23:47:26.084994 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.084881    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nwc5\" (UniqueName: \"kubernetes.io/projected/cf3f657a-f9d1-4579-b5ca-85813d334d85-kube-api-access-4nwc5\") pod \"cf3f657a-f9d1-4579-b5ca-85813d334d85\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") "
Jul 28 23:47:26.084994 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.084901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-tmp-dir\") pod \"cf3f657a-f9d1-4579-b5ca-85813d334d85\" (UID: \"cf3f657a-f9d1-4579-b5ca-85813d334d85\") "
Jul 28 23:47:26.085189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.084999    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf3f657a-f9d1-4579-b5ca-85813d334d85" (UID: "cf3f657a-f9d1-4579-b5ca-85813d334d85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:26.085189 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.085025    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-home" (OuterVolumeSpecName: "home") pod "cf3f657a-f9d1-4579-b5ca-85813d334d85" (UID: "cf3f657a-f9d1-4579-b5ca-85813d334d85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:26.085285 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.085200    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf3f657a-f9d1-4579-b5ca-85813d334d85" (UID: "cf3f657a-f9d1-4579-b5ca-85813d334d85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:26.085359 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.085333    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:26.085463 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.085362    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:26.087037 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.087010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3f657a-f9d1-4579-b5ca-85813d334d85-kube-api-access-4nwc5" (OuterVolumeSpecName: "kube-api-access-4nwc5") pod "cf3f657a-f9d1-4579-b5ca-85813d334d85" (UID: "cf3f657a-f9d1-4579-b5ca-85813d334d85"). InnerVolumeSpecName "kube-api-access-4nwc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:47:26.087450 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.087428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-dshm" (OuterVolumeSpecName: "dshm") pod "cf3f657a-f9d1-4579-b5ca-85813d334d85" (UID: "cf3f657a-f9d1-4579-b5ca-85813d334d85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:47:26.087517 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.087424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3f657a-f9d1-4579-b5ca-85813d334d85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf3f657a-f9d1-4579-b5ca-85813d334d85" (UID: "cf3f657a-f9d1-4579-b5ca-85813d334d85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:47:26.186534 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.186491    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3f657a-f9d1-4579-b5ca-85813d334d85-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:26.186534 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.186530    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nwc5\" (UniqueName: \"kubernetes.io/projected/cf3f657a-f9d1-4579-b5ca-85813d334d85-kube-api-access-4nwc5\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:26.186534 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.186541    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:26.186534 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.186551    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf3f657a-f9d1-4579-b5ca-85813d334d85-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:47:26.313921 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.313889    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:47:26.969433 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.969397    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"
Jul 28 23:47:26.969912 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.969433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65" event={"ID":"cf3f657a-f9d1-4579-b5ca-85813d334d85","Type":"ContainerDied","Data":"44bebe9964a4a99f69f78392e3e02abb71ac5b7249391f31a2681251d2ce34f3"}
Jul 28 23:47:26.969912 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.969493    2581 scope.go:117] "RemoveContainer" containerID="62317497fc1ac73f7845de4f008ed27364ce0eadf6c2b2c7fc4fc36b7fe9381e"
Jul 28 23:47:26.982736 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.982713    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7426ff66-6891-4ebe-9e08-a832111977f2" path="/var/lib/kubelet/pods/7426ff66-6891-4ebe-9e08-a832111977f2/volumes"
Jul 28 23:47:26.989209 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.989184    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"]
Jul 28 23:47:26.992956 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:26.992934    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bfdf0414-kserve-7cc4fdbcdc-4fp65"]
Jul 28 23:47:28.980791 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:28.980751    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3f657a-f9d1-4579-b5ca-85813d334d85" path="/var/lib/kubelet/pods/cf3f657a-f9d1-4579-b5ca-85813d334d85/volumes"
Jul 28 23:47:34.411576 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:34.411541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:47:50.729312 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:47:50.729275    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:48:00.617312 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:48:00.617274    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:48:10.512930 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:48:10.512833    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:48:20.410348 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:48:20.410310    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:48:29.819029 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:48:29.818992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:48:39.911145 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:48:39.911056    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:48:54.010801 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:48:54.010764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:49:09.921821 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:49:09.921790    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:49:22.620278 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:49:22.620243    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:49:30.714650 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:49:30.714595    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:49:45.413349 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:49:45.413314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:49:53.709839 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:49:53.709807    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:50:05.958860 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:50:05.958819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:50:18.216042 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:50:18.215998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:50:27.708705 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:50:27.708670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:50:42.714021 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:50:42.713988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:50:56.810639 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:50:56.810590    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:51:10.217652 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:51:10.217125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:51:39.124558 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:51:39.124465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:51:50.231339 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:51:50.231300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:52:14.327073 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.327013    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:52:14.803465 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.803433    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"]
Jul 28 23:52:14.803823 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.803809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7426ff66-6891-4ebe-9e08-a832111977f2" containerName="main"
Jul 28 23:52:14.803879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.803825    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7426ff66-6891-4ebe-9e08-a832111977f2" containerName="main"
Jul 28 23:52:14.803879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.803851    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3f657a-f9d1-4579-b5ca-85813d334d85" containerName="main"
Jul 28 23:52:14.803879 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.803857    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3f657a-f9d1-4579-b5ca-85813d334d85" containerName="main"
Jul 28 23:52:14.803989 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.803915    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7426ff66-6891-4ebe-9e08-a832111977f2" containerName="main"
Jul 28 23:52:14.803989 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.803925    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3f657a-f9d1-4579-b5ca-85813d334d85" containerName="main"
Jul 28 23:52:14.807002 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.806981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:14.809695 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.809673    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2aaf48a6\""
Jul 28 23:52:14.809783 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.809726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2aaf48a6-dockercfg-qw68q\""
Jul 28 23:52:14.817775 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.817752    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"]
Jul 28 23:52:14.874681 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.874647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d549f9c-da04-4c23-8bb3-3e66bad969b0-plugins-config-volume\") pod \"payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") " pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:14.874818 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.874691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcb7c\" (UniqueName: \"kubernetes.io/projected/9d549f9c-da04-4c23-8bb3-3e66bad969b0-kube-api-access-pcb7c\") pod \"payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") " pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:14.975516 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.975473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d549f9c-da04-4c23-8bb3-3e66bad969b0-plugins-config-volume\") pod \"payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") " pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:14.975516 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.975524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcb7c\" (UniqueName: \"kubernetes.io/projected/9d549f9c-da04-4c23-8bb3-3e66bad969b0-kube-api-access-pcb7c\") pod \"payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") " pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:14.976429 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.976404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d549f9c-da04-4c23-8bb3-3e66bad969b0-plugins-config-volume\") pod \"payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") " pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:14.991043 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:14.991011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcb7c\" (UniqueName: \"kubernetes.io/projected/9d549f9c-da04-4c23-8bb3-3e66bad969b0-kube-api-access-pcb7c\") pod \"payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") " pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:15.117276 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:15.117245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:15.454233 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:15.454209    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"]
Jul 28 23:52:15.456481 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:52:15.456455    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d549f9c_da04_4c23_8bb3_3e66bad969b0.slice/crio-e321d94e11fed7086fb6fdc22daa3666eb51caa0b42f64cb20a337ea05ed8b3e WatchSource:0}: Error finding container e321d94e11fed7086fb6fdc22daa3666eb51caa0b42f64cb20a337ea05ed8b3e: Status 404 returned error can't find the container with id e321d94e11fed7086fb6fdc22daa3666eb51caa0b42f64cb20a337ea05ed8b3e
Jul 28 23:52:15.458351 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:15.458337    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 23:52:16.290556 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:16.290518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v" event={"ID":"9d549f9c-da04-4c23-8bb3-3e66bad969b0","Type":"ContainerStarted","Data":"e321d94e11fed7086fb6fdc22daa3666eb51caa0b42f64cb20a337ea05ed8b3e"}
Jul 28 23:52:18.302779 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:18.302745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v" event={"ID":"9d549f9c-da04-4c23-8bb3-3e66bad969b0","Type":"ContainerStarted","Data":"1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515"}
Jul 28 23:52:18.303261 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:18.302880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:18.320924 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:18.320846    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v" podStartSLOduration=2.160923047 podStartE2EDuration="4.320831618s" totalImagesPullingTime="2.159908571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:52:18.319326483 +0000 UTC m=+1833.821873015" watchObservedRunningTime="2026-07-28 23:52:18.320831618 +0000 UTC m=+1833.823378149"
Jul 28 23:52:26.037668 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:26.037612    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:52:29.310053 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:29.310022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:54.860465 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:54.860425    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"]
Jul 28 23:52:54.861023 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:54.860704    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v" podUID="9d549f9c-da04-4c23-8bb3-3e66bad969b0" containerName="payload-processing" containerID="cri-o://1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515" gracePeriod=30
Jul 28 23:52:55.116347 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.116280    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:55.137406 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.137364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d549f9c-da04-4c23-8bb3-3e66bad969b0-plugins-config-volume\") pod \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") "
Jul 28 23:52:55.137546 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.137420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcb7c\" (UniqueName: \"kubernetes.io/projected/9d549f9c-da04-4c23-8bb3-3e66bad969b0-kube-api-access-pcb7c\") pod \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\" (UID: \"9d549f9c-da04-4c23-8bb3-3e66bad969b0\") "
Jul 28 23:52:55.137740 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.137719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d549f9c-da04-4c23-8bb3-3e66bad969b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d549f9c-da04-4c23-8bb3-3e66bad969b0" (UID: "9d549f9c-da04-4c23-8bb3-3e66bad969b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:52:55.139729 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.139598    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d549f9c-da04-4c23-8bb3-3e66bad969b0-kube-api-access-pcb7c" (OuterVolumeSpecName: "kube-api-access-pcb7c") pod "9d549f9c-da04-4c23-8bb3-3e66bad969b0" (UID: "9d549f9c-da04-4c23-8bb3-3e66bad969b0"). InnerVolumeSpecName "kube-api-access-pcb7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:52:55.238340 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.238301    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d549f9c-da04-4c23-8bb3-3e66bad969b0-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:52:55.238340 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.238331    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcb7c\" (UniqueName: \"kubernetes.io/projected/9d549f9c-da04-4c23-8bb3-3e66bad969b0-kube-api-access-pcb7c\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:52:55.478603 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.478495    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d549f9c-da04-4c23-8bb3-3e66bad969b0" containerID="1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515" exitCode=0
Jul 28 23:52:55.478603 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.478544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v" event={"ID":"9d549f9c-da04-4c23-8bb3-3e66bad969b0","Type":"ContainerDied","Data":"1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515"}
Jul 28 23:52:55.478603 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.478553    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"
Jul 28 23:52:55.479197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.478650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v" event={"ID":"9d549f9c-da04-4c23-8bb3-3e66bad969b0","Type":"ContainerDied","Data":"e321d94e11fed7086fb6fdc22daa3666eb51caa0b42f64cb20a337ea05ed8b3e"}
Jul 28 23:52:55.479197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.478680    2581 scope.go:117] "RemoveContainer" containerID="1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515"
Jul 28 23:52:55.487434 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.487416    2581 scope.go:117] "RemoveContainer" containerID="1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515"
Jul 28 23:52:55.487712 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:52:55.487691    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515\": container with ID starting with 1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515 not found: ID does not exist" containerID="1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515"
Jul 28 23:52:55.487782 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.487724    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515"} err="failed to get container status \"1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515\": rpc error: code = NotFound desc = could not find container \"1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515\": container with ID starting with 1bf9d33e0d0fce607cdbe90dbde4db9fb8298a22eb4e793b57b77130c6008515 not found: ID does not exist"
Jul 28 23:52:55.499293 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.499265    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"]
Jul 28 23:52:55.504335 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:55.504309    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2aaf48a6-7c848455db-l7p6v"]
Jul 28 23:52:56.981273 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:56.981242    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d549f9c-da04-4c23-8bb3-3e66bad969b0" path="/var/lib/kubelet/pods/9d549f9c-da04-4c23-8bb3-3e66bad969b0/volumes"
Jul 28 23:52:58.817808 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:58.817771    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:52:59.712562 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.712524    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"]
Jul 28 23:52:59.712907 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.712892    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d549f9c-da04-4c23-8bb3-3e66bad969b0" containerName="payload-processing"
Jul 28 23:52:59.712952 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.712908    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d549f9c-da04-4c23-8bb3-3e66bad969b0" containerName="payload-processing"
Jul 28 23:52:59.712990 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.712981    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d549f9c-da04-4c23-8bb3-3e66bad969b0" containerName="payload-processing"
Jul 28 23:52:59.715461 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.715444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:52:59.718073 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.718048    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-765c35ee\""
Jul 28 23:52:59.718326 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.718310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-765c35ee-dockercfg-6f5pp\""
Jul 28 23:52:59.776755 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.776724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1605b1-7682-4d3a-807b-684891aaf2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:52:59.776917 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.776808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjf5\" (UniqueName: \"kubernetes.io/projected/af1605b1-7682-4d3a-807b-684891aaf2fa-kube-api-access-6hjf5\") pod \"payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:52:59.821511 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.821472    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"]
Jul 28 23:52:59.823114 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.823092    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"]
Jul 28 23:52:59.825300 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.825284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:52:59.842726 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.842695    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"]
Jul 28 23:52:59.877310 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.877280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjf5\" (UniqueName: \"kubernetes.io/projected/af1605b1-7682-4d3a-807b-684891aaf2fa-kube-api-access-6hjf5\") pod \"payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:52:59.877468 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.877346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1605b1-7682-4d3a-807b-684891aaf2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:52:59.877468 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.877400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkjll\" (UniqueName: \"kubernetes.io/projected/78078a86-dc03-4c91-8989-512958a1b1da-kube-api-access-qkjll\") pod \"payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") " pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:52:59.877468 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.877433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78078a86-dc03-4c91-8989-512958a1b1da-plugins-config-volume\") pod \"payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") " pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:52:59.878081 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.878061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1605b1-7682-4d3a-807b-684891aaf2fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:52:59.886639 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.886608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjf5\" (UniqueName: \"kubernetes.io/projected/af1605b1-7682-4d3a-807b-684891aaf2fa-kube-api-access-6hjf5\") pod \"payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:52:59.978806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.978714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkjll\" (UniqueName: \"kubernetes.io/projected/78078a86-dc03-4c91-8989-512958a1b1da-kube-api-access-qkjll\") pod \"payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") " pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:52:59.978806 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.978761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78078a86-dc03-4c91-8989-512958a1b1da-plugins-config-volume\") pod \"payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") " pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:52:59.979473 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.979454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78078a86-dc03-4c91-8989-512958a1b1da-plugins-config-volume\") pod \"payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") " pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:52:59.992951 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:52:59.992927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkjll\" (UniqueName: \"kubernetes.io/projected/78078a86-dc03-4c91-8989-512958a1b1da-kube-api-access-qkjll\") pod \"payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") " pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:53:00.025793 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.025766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:53:00.135092 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.135058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:53:00.156710 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.156685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"]
Jul 28 23:53:00.158512 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:53:00.158487    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1605b1_7682_4d3a_807b_684891aaf2fa.slice/crio-c5e8c259a577d5154d1db27b5612cee2b132d5b9bc0e783ed90586bb44761a9c WatchSource:0}: Error finding container c5e8c259a577d5154d1db27b5612cee2b132d5b9bc0e783ed90586bb44761a9c: Status 404 returned error can't find the container with id c5e8c259a577d5154d1db27b5612cee2b132d5b9bc0e783ed90586bb44761a9c
Jul 28 23:53:00.271650 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.271607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"]
Jul 28 23:53:00.273469 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:53:00.273440    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78078a86_dc03_4c91_8989_512958a1b1da.slice/crio-fde9385b047902ee2b3e95c732d290da3e7cbaf0a6e4c36de189437b8fe008de WatchSource:0}: Error finding container fde9385b047902ee2b3e95c732d290da3e7cbaf0a6e4c36de189437b8fe008de: Status 404 returned error can't find the container with id fde9385b047902ee2b3e95c732d290da3e7cbaf0a6e4c36de189437b8fe008de
Jul 28 23:53:00.510168 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.510122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j" event={"ID":"78078a86-dc03-4c91-8989-512958a1b1da","Type":"ContainerStarted","Data":"c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff"}
Jul 28 23:53:00.510502 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.510211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j" event={"ID":"78078a86-dc03-4c91-8989-512958a1b1da","Type":"ContainerStarted","Data":"fde9385b047902ee2b3e95c732d290da3e7cbaf0a6e4c36de189437b8fe008de"}
Jul 28 23:53:00.510502 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.510236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:53:00.511614 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.511588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz" event={"ID":"af1605b1-7682-4d3a-807b-684891aaf2fa","Type":"ContainerStarted","Data":"0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8"}
Jul 28 23:53:00.511754 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.511650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz" event={"ID":"af1605b1-7682-4d3a-807b-684891aaf2fa","Type":"ContainerStarted","Data":"c5e8c259a577d5154d1db27b5612cee2b132d5b9bc0e783ed90586bb44761a9c"}
Jul 28 23:53:00.511754 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.511688    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:53:00.528570 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.528490    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j" podStartSLOduration=1.528478773 podStartE2EDuration="1.528478773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:53:00.527302649 +0000 UTC m=+1876.029849183" watchObservedRunningTime="2026-07-28 23:53:00.528478773 +0000 UTC m=+1876.031025304"
Jul 28 23:53:00.543448 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:00.543409    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz" podStartSLOduration=1.543394157 podStartE2EDuration="1.543394157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:53:00.541472568 +0000 UTC m=+1876.044019101" watchObservedRunningTime="2026-07-28 23:53:00.543394157 +0000 UTC m=+1876.045940722"
Jul 28 23:53:09.022072 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:09.021996    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:53:11.518267 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:11.518232    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:53:11.520709 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:11.518398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:53:29.931968 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:29.931932    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"]
Jul 28 23:53:29.932616 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:29.932173    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j" podUID="78078a86-dc03-4c91-8989-512958a1b1da" containerName="payload-processing" containerID="cri-o://c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff" gracePeriod=30
Jul 28 23:53:29.937116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:29.937090    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"]
Jul 28 23:53:29.938257 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:29.938223    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz" podUID="af1605b1-7682-4d3a-807b-684891aaf2fa" containerName="payload-pre-processing" containerID="cri-o://0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8" gracePeriod=30
Jul 28 23:53:30.201506 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.201484    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:53:30.205374 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.205348    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:53:30.360323 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.360285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78078a86-dc03-4c91-8989-512958a1b1da-plugins-config-volume\") pod \"78078a86-dc03-4c91-8989-512958a1b1da\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") "
Jul 28 23:53:30.360528 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.360412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hjf5\" (UniqueName: \"kubernetes.io/projected/af1605b1-7682-4d3a-807b-684891aaf2fa-kube-api-access-6hjf5\") pod \"af1605b1-7682-4d3a-807b-684891aaf2fa\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") "
Jul 28 23:53:30.360528 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.360441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1605b1-7682-4d3a-807b-684891aaf2fa-plugins-config-volume\") pod \"af1605b1-7682-4d3a-807b-684891aaf2fa\" (UID: \"af1605b1-7682-4d3a-807b-684891aaf2fa\") "
Jul 28 23:53:30.360528 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.360463    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkjll\" (UniqueName: \"kubernetes.io/projected/78078a86-dc03-4c91-8989-512958a1b1da-kube-api-access-qkjll\") pod \"78078a86-dc03-4c91-8989-512958a1b1da\" (UID: \"78078a86-dc03-4c91-8989-512958a1b1da\") "
Jul 28 23:53:30.360770 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.360746    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78078a86-dc03-4c91-8989-512958a1b1da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78078a86-dc03-4c91-8989-512958a1b1da" (UID: "78078a86-dc03-4c91-8989-512958a1b1da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:53:30.360820 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.360804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1605b1-7682-4d3a-807b-684891aaf2fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af1605b1-7682-4d3a-807b-684891aaf2fa" (UID: "af1605b1-7682-4d3a-807b-684891aaf2fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:53:30.362604 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.362577    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1605b1-7682-4d3a-807b-684891aaf2fa-kube-api-access-6hjf5" (OuterVolumeSpecName: "kube-api-access-6hjf5") pod "af1605b1-7682-4d3a-807b-684891aaf2fa" (UID: "af1605b1-7682-4d3a-807b-684891aaf2fa"). InnerVolumeSpecName "kube-api-access-6hjf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:53:30.362720 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.362615    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78078a86-dc03-4c91-8989-512958a1b1da-kube-api-access-qkjll" (OuterVolumeSpecName: "kube-api-access-qkjll") pod "78078a86-dc03-4c91-8989-512958a1b1da" (UID: "78078a86-dc03-4c91-8989-512958a1b1da"). InnerVolumeSpecName "kube-api-access-qkjll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:53:30.461765 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.461688    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78078a86-dc03-4c91-8989-512958a1b1da-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:53:30.461765 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.461714    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hjf5\" (UniqueName: \"kubernetes.io/projected/af1605b1-7682-4d3a-807b-684891aaf2fa-kube-api-access-6hjf5\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:53:30.461765 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.461726    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1605b1-7682-4d3a-807b-684891aaf2fa-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:53:30.461765 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.461735    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkjll\" (UniqueName: \"kubernetes.io/projected/78078a86-dc03-4c91-8989-512958a1b1da-kube-api-access-qkjll\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:53:30.655045 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.655011    2581 generic.go:356] "Generic (PLEG): container finished" podID="af1605b1-7682-4d3a-807b-684891aaf2fa" containerID="0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8" exitCode=0
Jul 28 23:53:30.655239 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.655053    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"
Jul 28 23:53:30.655239 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.655106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz" event={"ID":"af1605b1-7682-4d3a-807b-684891aaf2fa","Type":"ContainerDied","Data":"0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8"}
Jul 28 23:53:30.655239 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.655158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz" event={"ID":"af1605b1-7682-4d3a-807b-684891aaf2fa","Type":"ContainerDied","Data":"c5e8c259a577d5154d1db27b5612cee2b132d5b9bc0e783ed90586bb44761a9c"}
Jul 28 23:53:30.655239 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.655190    2581 scope.go:117] "RemoveContainer" containerID="0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8"
Jul 28 23:53:30.656456 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.656434    2581 generic.go:356] "Generic (PLEG): container finished" podID="78078a86-dc03-4c91-8989-512958a1b1da" containerID="c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff" exitCode=0
Jul 28 23:53:30.656553 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.656479    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"
Jul 28 23:53:30.656553 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.656522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j" event={"ID":"78078a86-dc03-4c91-8989-512958a1b1da","Type":"ContainerDied","Data":"c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff"}
Jul 28 23:53:30.656673 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.656574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j" event={"ID":"78078a86-dc03-4c91-8989-512958a1b1da","Type":"ContainerDied","Data":"fde9385b047902ee2b3e95c732d290da3e7cbaf0a6e4c36de189437b8fe008de"}
Jul 28 23:53:30.666651 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.666612    2581 scope.go:117] "RemoveContainer" containerID="0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8"
Jul 28 23:53:30.667035 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:53:30.666999    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8\": container with ID starting with 0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8 not found: ID does not exist" containerID="0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8"
Jul 28 23:53:30.667129 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.667046    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8"} err="failed to get container status \"0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8\": rpc error: code = NotFound desc = could not find container \"0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8\": container with ID starting with 0ae551f9e23bc7d2d0aca6ccc4cfacfaf33f3437bbeb648f5f80bf4ceb6acfe8 not found: ID does not exist"
Jul 28 23:53:30.667129 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.667070    2581 scope.go:117] "RemoveContainer" containerID="c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff"
Jul 28 23:53:30.681264 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.681230    2581 scope.go:117] "RemoveContainer" containerID="c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff"
Jul 28 23:53:30.681678 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:53:30.681656    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff\": container with ID starting with c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff not found: ID does not exist" containerID="c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff"
Jul 28 23:53:30.681766 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.681682    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff"} err="failed to get container status \"c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff\": rpc error: code = NotFound desc = could not find container \"c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff\": container with ID starting with c577358519f97ed81344c5a1bb1a5f02e6cfd313d4ab5db426fcbbf9bd1b75ff not found: ID does not exist"
Jul 28 23:53:30.683228 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.683206    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"]
Jul 28 23:53:30.686076 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.686053    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-765c35ee-585cf7c956-k2dcz"]
Jul 28 23:53:30.695953 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.695934    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"]
Jul 28 23:53:30.698957 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.698939    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-765c35ee-64b4c669fd-4n69j"]
Jul 28 23:53:30.982122 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.982091    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78078a86-dc03-4c91-8989-512958a1b1da" path="/var/lib/kubelet/pods/78078a86-dc03-4c91-8989-512958a1b1da/volumes"
Jul 28 23:53:30.982444 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:30.982425    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af1605b1-7682-4d3a-807b-684891aaf2fa" path="/var/lib/kubelet/pods/af1605b1-7682-4d3a-807b-684891aaf2fa/volumes"
Jul 28 23:53:33.822364 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:53:33.822321    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:54:55.314475 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:54:55.314394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:55:12.411900 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:12.411865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:55:22.316021 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:22.315986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:55:28.033498 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.033457    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"]
Jul 28 23:55:28.034016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.033821    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78078a86-dc03-4c91-8989-512958a1b1da" containerName="payload-processing"
Jul 28 23:55:28.034016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.033833    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="78078a86-dc03-4c91-8989-512958a1b1da" containerName="payload-processing"
Jul 28 23:55:28.034016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.033842    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af1605b1-7682-4d3a-807b-684891aaf2fa" containerName="payload-pre-processing"
Jul 28 23:55:28.034016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.033848    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af1605b1-7682-4d3a-807b-684891aaf2fa" containerName="payload-pre-processing"
Jul 28 23:55:28.034016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.033932    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="78078a86-dc03-4c91-8989-512958a1b1da" containerName="payload-processing"
Jul 28 23:55:28.034016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.033942    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af1605b1-7682-4d3a-807b-684891aaf2fa" containerName="payload-pre-processing"
Jul 28 23:55:28.036881 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.036863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.039266 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.039238    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-790b15df\""
Jul 28 23:55:28.047904 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.047872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"]
Jul 28 23:55:28.089387 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.089350    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"]
Jul 28 23:55:28.092073 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.092048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rhx\" (UniqueName: \"kubernetes.io/projected/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-kube-api-access-25rhx\") pod \"payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.092201 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.092118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.092703 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.092687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.095225 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.095206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-790b15df-dockercfg-lmx24\""
Jul 28 23:55:28.104097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.104071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"]
Jul 28 23:55:28.192926 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.192875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rhx\" (UniqueName: \"kubernetes.io/projected/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-kube-api-access-25rhx\") pod \"payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.193125 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.192970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.193125 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.192998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2w8\" (UniqueName: \"kubernetes.io/projected/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-kube-api-access-cq2w8\") pod \"payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") " pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.193125 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.193030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-plugins-config-volume\") pod \"payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") " pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.193731 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.193710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.200341 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.200318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rhx\" (UniqueName: \"kubernetes.io/projected/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-kube-api-access-25rhx\") pod \"payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.294072 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.293977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-plugins-config-volume\") pod \"payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") " pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.294225 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.294091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2w8\" (UniqueName: \"kubernetes.io/projected/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-kube-api-access-cq2w8\") pod \"payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") " pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.294745 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.294726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-plugins-config-volume\") pod \"payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") " pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.301872 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.301845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2w8\" (UniqueName: \"kubernetes.io/projected/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-kube-api-access-cq2w8\") pod \"payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") " pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.347970 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.347935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:28.403369 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.403337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:28.491440 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.491414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"]
Jul 28 23:55:28.493527 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:55:28.493499    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750cc0d0_d8a5_4cd5_baee_81a7191c85bc.slice/crio-0223e9397204d0a85eca6d8004eefe8f3fec26f599f1334176ee30fa6acfaf6f WatchSource:0}: Error finding container 0223e9397204d0a85eca6d8004eefe8f3fec26f599f1334176ee30fa6acfaf6f: Status 404 returned error can't find the container with id 0223e9397204d0a85eca6d8004eefe8f3fec26f599f1334176ee30fa6acfaf6f
Jul 28 23:55:28.545454 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:28.545426    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"]
Jul 28 23:55:28.548715 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:55:28.548690    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08cca4a_8ba1_4f62_aac9_f9c37ea36341.slice/crio-39733606b859ce61d062ffc51ebc71ed1fdf0640e5630e8818cc4cf30229e3e9 WatchSource:0}: Error finding container 39733606b859ce61d062ffc51ebc71ed1fdf0640e5630e8818cc4cf30229e3e9: Status 404 returned error can't find the container with id 39733606b859ce61d062ffc51ebc71ed1fdf0640e5630e8818cc4cf30229e3e9
Jul 28 23:55:29.211279 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.211230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4" event={"ID":"b08cca4a-8ba1-4f62-aac9-f9c37ea36341","Type":"ContainerStarted","Data":"d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303"}
Jul 28 23:55:29.211760 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.211302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4" event={"ID":"b08cca4a-8ba1-4f62-aac9-f9c37ea36341","Type":"ContainerStarted","Data":"39733606b859ce61d062ffc51ebc71ed1fdf0640e5630e8818cc4cf30229e3e9"}
Jul 28 23:55:29.211760 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.211336    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:55:29.212919 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.212893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd" event={"ID":"750cc0d0-d8a5-4cd5-baee-81a7191c85bc","Type":"ContainerStarted","Data":"c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a"}
Jul 28 23:55:29.213029 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.212936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd" event={"ID":"750cc0d0-d8a5-4cd5-baee-81a7191c85bc","Type":"ContainerStarted","Data":"0223e9397204d0a85eca6d8004eefe8f3fec26f599f1334176ee30fa6acfaf6f"}
Jul 28 23:55:29.213029 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.212959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:29.229042 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.228988    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4" podStartSLOduration=1.228972133 podStartE2EDuration="1.228972133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:55:29.226935676 +0000 UTC m=+2024.729482209" watchObservedRunningTime="2026-07-28 23:55:29.228972133 +0000 UTC m=+2024.731518663"
Jul 28 23:55:29.241254 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:29.241191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd" podStartSLOduration=1.241168118 podStartE2EDuration="1.241168118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:55:29.240253557 +0000 UTC m=+2024.742800083" watchObservedRunningTime="2026-07-28 23:55:29.241168118 +0000 UTC m=+2024.743714650"
Jul 28 23:55:32.315974 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:32.315938    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:55:40.220053 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:40.220019    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:55:40.220543 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:55:40.220092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:56:09.639529 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:09.639438    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"]
Jul 28 23:56:09.640131 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:09.639717    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4" podUID="b08cca4a-8ba1-4f62-aac9-f9c37ea36341" containerName="payload-processing" containerID="cri-o://d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303" gracePeriod=30
Jul 28 23:56:09.647878 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:09.647848    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"]
Jul 28 23:56:09.648112 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:09.648086    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd" podUID="750cc0d0-d8a5-4cd5-baee-81a7191c85bc" containerName="payload-pre-processing" containerID="cri-o://c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a" gracePeriod=30
Jul 28 23:56:09.894047 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:09.893988    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:56:09.926621 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:09.926594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:56:10.057939 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.057901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-plugins-config-volume\") pod \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") "
Jul 28 23:56:10.058116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.057961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25rhx\" (UniqueName: \"kubernetes.io/projected/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-kube-api-access-25rhx\") pod \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\" (UID: \"750cc0d0-d8a5-4cd5-baee-81a7191c85bc\") "
Jul 28 23:56:10.058116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.057977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq2w8\" (UniqueName: \"kubernetes.io/projected/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-kube-api-access-cq2w8\") pod \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") "
Jul 28 23:56:10.058116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.058001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-plugins-config-volume\") pod \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\" (UID: \"b08cca4a-8ba1-4f62-aac9-f9c37ea36341\") "
Jul 28 23:56:10.058460 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.058427    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b08cca4a-8ba1-4f62-aac9-f9c37ea36341" (UID: "b08cca4a-8ba1-4f62-aac9-f9c37ea36341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:56:10.058545 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.058464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "750cc0d0-d8a5-4cd5-baee-81a7191c85bc" (UID: "750cc0d0-d8a5-4cd5-baee-81a7191c85bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:56:10.060119 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.060093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-kube-api-access-25rhx" (OuterVolumeSpecName: "kube-api-access-25rhx") pod "750cc0d0-d8a5-4cd5-baee-81a7191c85bc" (UID: "750cc0d0-d8a5-4cd5-baee-81a7191c85bc"). InnerVolumeSpecName "kube-api-access-25rhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:56:10.060214 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.060117    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-kube-api-access-cq2w8" (OuterVolumeSpecName: "kube-api-access-cq2w8") pod "b08cca4a-8ba1-4f62-aac9-f9c37ea36341" (UID: "b08cca4a-8ba1-4f62-aac9-f9c37ea36341"). InnerVolumeSpecName "kube-api-access-cq2w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:56:10.159294 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.159217    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:56:10.159294 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.159245    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25rhx\" (UniqueName: \"kubernetes.io/projected/750cc0d0-d8a5-4cd5-baee-81a7191c85bc-kube-api-access-25rhx\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:56:10.159294 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.159255    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2w8\" (UniqueName: \"kubernetes.io/projected/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-kube-api-access-cq2w8\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:56:10.159294 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.159264    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08cca4a-8ba1-4f62-aac9-f9c37ea36341-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:56:10.401763 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.401730    2581 generic.go:356] "Generic (PLEG): container finished" podID="b08cca4a-8ba1-4f62-aac9-f9c37ea36341" containerID="d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303" exitCode=0
Jul 28 23:56:10.401947 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.401778    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"
Jul 28 23:56:10.401947 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.401817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4" event={"ID":"b08cca4a-8ba1-4f62-aac9-f9c37ea36341","Type":"ContainerDied","Data":"d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303"}
Jul 28 23:56:10.401947 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.401875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4" event={"ID":"b08cca4a-8ba1-4f62-aac9-f9c37ea36341","Type":"ContainerDied","Data":"39733606b859ce61d062ffc51ebc71ed1fdf0640e5630e8818cc4cf30229e3e9"}
Jul 28 23:56:10.401947 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.401906    2581 scope.go:117] "RemoveContainer" containerID="d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303"
Jul 28 23:56:10.403115 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.403092    2581 generic.go:356] "Generic (PLEG): container finished" podID="750cc0d0-d8a5-4cd5-baee-81a7191c85bc" containerID="c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a" exitCode=0
Jul 28 23:56:10.403193 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.403138    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"
Jul 28 23:56:10.403242 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.403139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd" event={"ID":"750cc0d0-d8a5-4cd5-baee-81a7191c85bc","Type":"ContainerDied","Data":"c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a"}
Jul 28 23:56:10.403335 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.403270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd" event={"ID":"750cc0d0-d8a5-4cd5-baee-81a7191c85bc","Type":"ContainerDied","Data":"0223e9397204d0a85eca6d8004eefe8f3fec26f599f1334176ee30fa6acfaf6f"}
Jul 28 23:56:10.412641 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.412597    2581 scope.go:117] "RemoveContainer" containerID="d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303"
Jul 28 23:56:10.413002 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:56:10.412982    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303\": container with ID starting with d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303 not found: ID does not exist" containerID="d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303"
Jul 28 23:56:10.413108 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.413007    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303"} err="failed to get container status \"d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303\": rpc error: code = NotFound desc = could not find container \"d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303\": container with ID starting with d60363b788395d2ab11bb3b3510b466ee084c85007494eebff719544f1cc3303 not found: ID does not exist"
Jul 28 23:56:10.413108 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.413023    2581 scope.go:117] "RemoveContainer" containerID="c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a"
Jul 28 23:56:10.423449 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.423429    2581 scope.go:117] "RemoveContainer" containerID="c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a"
Jul 28 23:56:10.424121 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:56:10.424093    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a\": container with ID starting with c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a not found: ID does not exist" containerID="c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a"
Jul 28 23:56:10.424222 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.424129    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a"} err="failed to get container status \"c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a\": rpc error: code = NotFound desc = could not find container \"c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a\": container with ID starting with c380139616f9950f9f522fefa646d44b5d0bc5bf5fef04ded8083aa7f20e9d8a not found: ID does not exist"
Jul 28 23:56:10.427568 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.427486    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"]
Jul 28 23:56:10.430242 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.430222    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-790b15df-fb6cbb7cd-5fdw4"]
Jul 28 23:56:10.442270 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.442250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"]
Jul 28 23:56:10.450885 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.447702    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-790b15df-7d8d5f78dd-g84jd"]
Jul 28 23:56:10.981833 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.981803    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="750cc0d0-d8a5-4cd5-baee-81a7191c85bc" path="/var/lib/kubelet/pods/750cc0d0-d8a5-4cd5-baee-81a7191c85bc/volumes"
Jul 28 23:56:10.982220 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:10.982150    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08cca4a-8ba1-4f62-aac9-f9c37ea36341" path="/var/lib/kubelet/pods/b08cca4a-8ba1-4f62-aac9-f9c37ea36341/volumes"
Jul 28 23:56:12.517248 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:12.517212    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:56:39.253288 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.253253    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"]
Jul 28 23:56:39.253770 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.253593    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750cc0d0-d8a5-4cd5-baee-81a7191c85bc" containerName="payload-pre-processing"
Jul 28 23:56:39.253770 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.253604    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="750cc0d0-d8a5-4cd5-baee-81a7191c85bc" containerName="payload-pre-processing"
Jul 28 23:56:39.253770 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.253614    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08cca4a-8ba1-4f62-aac9-f9c37ea36341" containerName="payload-processing"
Jul 28 23:56:39.253770 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.253620    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08cca4a-8ba1-4f62-aac9-f9c37ea36341" containerName="payload-processing"
Jul 28 23:56:39.253770 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.253703    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08cca4a-8ba1-4f62-aac9-f9c37ea36341" containerName="payload-processing"
Jul 28 23:56:39.253770 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.253713    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="750cc0d0-d8a5-4cd5-baee-81a7191c85bc" containerName="payload-pre-processing"
Jul 28 23:56:39.256552 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.256535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.259253 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.259229    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-21449885-dockercfg-n2f79\""
Jul 28 23:56:39.259381 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.259253    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-21449885\""
Jul 28 23:56:39.266096 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.266070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"]
Jul 28 23:56:39.276236 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.276210    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"]
Jul 28 23:56:39.279391 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.279376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.288955 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.288931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"]
Jul 28 23:56:39.312950 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.312922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7493d4-c051-4d82-acf8-43f8270a5fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.313058 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.312958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnkbj\" (UniqueName: \"kubernetes.io/projected/9d7493d4-c051-4d82-acf8-43f8270a5fc5-kube-api-access-rnkbj\") pod \"payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.414203 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.414171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c8a763-fcd3-460e-a765-f91cbb73bc6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-21449885-7f446c98b6-svwhd\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") " pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.414363 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.414219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7493d4-c051-4d82-acf8-43f8270a5fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.414363 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.414239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpd99\" (UniqueName: \"kubernetes.io/projected/04c8a763-fcd3-460e-a765-f91cbb73bc6e-kube-api-access-dpd99\") pod \"payload-processing-e2e-mt-21449885-7f446c98b6-svwhd\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") " pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.414363 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.414267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnkbj\" (UniqueName: \"kubernetes.io/projected/9d7493d4-c051-4d82-acf8-43f8270a5fc5-kube-api-access-rnkbj\") pod \"payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.414918 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.414899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7493d4-c051-4d82-acf8-43f8270a5fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.422012 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.421995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnkbj\" (UniqueName: \"kubernetes.io/projected/9d7493d4-c051-4d82-acf8-43f8270a5fc5-kube-api-access-rnkbj\") pod \"payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.515197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.515120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpd99\" (UniqueName: \"kubernetes.io/projected/04c8a763-fcd3-460e-a765-f91cbb73bc6e-kube-api-access-dpd99\") pod \"payload-processing-e2e-mt-21449885-7f446c98b6-svwhd\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") " pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.515333 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.515214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c8a763-fcd3-460e-a765-f91cbb73bc6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-21449885-7f446c98b6-svwhd\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") " pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.515924 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.515905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c8a763-fcd3-460e-a765-f91cbb73bc6e-plugins-config-volume\") pod \"payload-processing-e2e-mt-21449885-7f446c98b6-svwhd\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") " pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.523120 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.523092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpd99\" (UniqueName: \"kubernetes.io/projected/04c8a763-fcd3-460e-a765-f91cbb73bc6e-kube-api-access-dpd99\") pod \"payload-processing-e2e-mt-21449885-7f446c98b6-svwhd\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") " pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.567029 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.566997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:39.590293 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.590266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:39.705552 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.705521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"]
Jul 28 23:56:39.708368 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:56:39.708325    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7493d4_c051_4d82_acf8_43f8270a5fc5.slice/crio-4e82871010aa971187ac1edb4c5d1b099dd73217ad1c933e22fbf20d2a670743 WatchSource:0}: Error finding container 4e82871010aa971187ac1edb4c5d1b099dd73217ad1c933e22fbf20d2a670743: Status 404 returned error can't find the container with id 4e82871010aa971187ac1edb4c5d1b099dd73217ad1c933e22fbf20d2a670743
Jul 28 23:56:39.734856 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:39.734827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"]
Jul 28 23:56:39.740950 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:56:39.740920    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c8a763_fcd3_460e_a765_f91cbb73bc6e.slice/crio-9fd51da0319d6c5c742ad8dd7fe2f7df3df2c1353c6f5eb9bf0ed25d84ab5173 WatchSource:0}: Error finding container 9fd51da0319d6c5c742ad8dd7fe2f7df3df2c1353c6f5eb9bf0ed25d84ab5173: Status 404 returned error can't find the container with id 9fd51da0319d6c5c742ad8dd7fe2f7df3df2c1353c6f5eb9bf0ed25d84ab5173
Jul 28 23:56:40.541533 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.541497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97" event={"ID":"9d7493d4-c051-4d82-acf8-43f8270a5fc5","Type":"ContainerStarted","Data":"76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857"}
Jul 28 23:56:40.542016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.541677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:56:40.542016 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.541712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97" event={"ID":"9d7493d4-c051-4d82-acf8-43f8270a5fc5","Type":"ContainerStarted","Data":"4e82871010aa971187ac1edb4c5d1b099dd73217ad1c933e22fbf20d2a670743"}
Jul 28 23:56:40.543045 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.543018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd" event={"ID":"04c8a763-fcd3-460e-a765-f91cbb73bc6e","Type":"ContainerStarted","Data":"2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d"}
Jul 28 23:56:40.543183 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.543062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd" event={"ID":"04c8a763-fcd3-460e-a765-f91cbb73bc6e","Type":"ContainerStarted","Data":"9fd51da0319d6c5c742ad8dd7fe2f7df3df2c1353c6f5eb9bf0ed25d84ab5173"}
Jul 28 23:56:40.543183 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.543101    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:40.556912 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.556864    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97" podStartSLOduration=1.5568464359999998 podStartE2EDuration="1.556846436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:56:40.5559453 +0000 UTC m=+2096.058491845" watchObservedRunningTime="2026-07-28 23:56:40.556846436 +0000 UTC m=+2096.059392969"
Jul 28 23:56:40.569871 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:40.569828    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd" podStartSLOduration=1.569811664 podStartE2EDuration="1.569811664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:56:40.569720027 +0000 UTC m=+2096.072266561" watchObservedRunningTime="2026-07-28 23:56:40.569811664 +0000 UTC m=+2096.072358196"
Jul 28 23:56:44.011825 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:44.011788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:56:51.550554 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:51.550517    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:56:51.550982 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:56:51.550685    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:57:12.798780 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.798745    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"]
Jul 28 23:57:12.802451 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.802430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:12.804834 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.804810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-21449885\"/\"openshift-service-ca.crt\""
Jul 28 23:57:12.805217 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.805199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-21449885\"/\"kube-root-ca.crt\""
Jul 28 23:57:12.806039 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.806022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-21449885\"/\"e2e-lifecycle-model-21449885-kserve-self-signed-certs\""
Jul 28 23:57:12.806197 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.806177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-21449885\"/\"default-dockercfg-bw9pl\""
Jul 28 23:57:12.817583 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.817561    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"]
Jul 28 23:57:12.917581 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.917541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-dshm\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:12.917790 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.917590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krn5k\" (UniqueName: \"kubernetes.io/projected/da19e146-ea96-415f-869d-85a29964cd9c-kube-api-access-krn5k\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:12.917790 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.917694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-tmp-dir\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:12.917790 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.917727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-model-cache\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:12.917790 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.917751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-home\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:12.917790 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:12.917782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da19e146-ea96-415f-869d-85a29964cd9c-tls-certs\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.018809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.018756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-dshm\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.018809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.018814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krn5k\" (UniqueName: \"kubernetes.io/projected/da19e146-ea96-415f-869d-85a29964cd9c-kube-api-access-krn5k\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.019080 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.018862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-tmp-dir\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.019080 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.018880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-model-cache\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.019080 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.018899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-home\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.019260 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.019171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da19e146-ea96-415f-869d-85a29964cd9c-tls-certs\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.019336 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.019313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-model-cache\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.019485 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.019454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-home\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.019621 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.019503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-tmp-dir\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.021134 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.021114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-dshm\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.021423 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.021404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da19e146-ea96-415f-869d-85a29964cd9c-tls-certs\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.026890 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.026868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krn5k\" (UniqueName: \"kubernetes.io/projected/da19e146-ea96-415f-869d-85a29964cd9c-kube-api-access-krn5k\") pod \"e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") " pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.111856 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.111826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:13.240344 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.240315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"]
Jul 28 23:57:13.242059 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:57:13.242035    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda19e146_ea96_415f_869d_85a29964cd9c.slice/crio-37514e88b7d09984259e06657cb1154274a8daf7fbbd5305411b4a3862bc138d WatchSource:0}: Error finding container 37514e88b7d09984259e06657cb1154274a8daf7fbbd5305411b4a3862bc138d: Status 404 returned error can't find the container with id 37514e88b7d09984259e06657cb1154274a8daf7fbbd5305411b4a3862bc138d
Jul 28 23:57:13.698922 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.698889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf" event={"ID":"da19e146-ea96-415f-869d-85a29964cd9c","Type":"ContainerStarted","Data":"61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83"}
Jul 28 23:57:13.698922 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.698931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf" event={"ID":"da19e146-ea96-415f-869d-85a29964cd9c","Type":"ContainerStarted","Data":"37514e88b7d09984259e06657cb1154274a8daf7fbbd5305411b4a3862bc138d"}
Jul 28 23:57:13.714659 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:13.714595    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf" podStartSLOduration=1.714580008 podStartE2EDuration="1.714580008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:57:13.713494176 +0000 UTC m=+2129.216040707" watchObservedRunningTime="2026-07-28 23:57:13.714580008 +0000 UTC m=+2129.217126538"
Jul 28 23:57:15.813822 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:15.813785    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:57:23.112116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:23.112053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:23.112116 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:23.112134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:23.124795 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:23.124766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:23.757724 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:23.757696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:57:33.310737 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:33.310701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:57:52.108533 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:52.108451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:57:53.134095 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.134061    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"]
Jul 28 23:57:53.134570 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.134284    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd" podUID="04c8a763-fcd3-460e-a765-f91cbb73bc6e" containerName="payload-processing" containerID="cri-o://2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d" gracePeriod=30
Jul 28 23:57:53.142853 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.142172    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"]
Jul 28 23:57:53.142853 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.142674    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97" podUID="9d7493d4-c051-4d82-acf8-43f8270a5fc5" containerName="payload-pre-processing" containerID="cri-o://76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857" gracePeriod=30
Jul 28 23:57:53.419263 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.419236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:57:53.422159 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.422141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:57:53.485899 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.485865    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c8a763-fcd3-460e-a765-f91cbb73bc6e-plugins-config-volume\") pod \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") "
Jul 28 23:57:53.486079 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.485927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpd99\" (UniqueName: \"kubernetes.io/projected/04c8a763-fcd3-460e-a765-f91cbb73bc6e-kube-api-access-dpd99\") pod \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\" (UID: \"04c8a763-fcd3-460e-a765-f91cbb73bc6e\") "
Jul 28 23:57:53.486079 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.485961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7493d4-c051-4d82-acf8-43f8270a5fc5-plugins-config-volume\") pod \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") "
Jul 28 23:57:53.486079 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.486041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnkbj\" (UniqueName: \"kubernetes.io/projected/9d7493d4-c051-4d82-acf8-43f8270a5fc5-kube-api-access-rnkbj\") pod \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\" (UID: \"9d7493d4-c051-4d82-acf8-43f8270a5fc5\") "
Jul 28 23:57:53.486373 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.486345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c8a763-fcd3-460e-a765-f91cbb73bc6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04c8a763-fcd3-460e-a765-f91cbb73bc6e" (UID: "04c8a763-fcd3-460e-a765-f91cbb73bc6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:57:53.486429 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.486381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7493d4-c051-4d82-acf8-43f8270a5fc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d7493d4-c051-4d82-acf8-43f8270a5fc5" (UID: "9d7493d4-c051-4d82-acf8-43f8270a5fc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 23:57:53.488058 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.488035    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7493d4-c051-4d82-acf8-43f8270a5fc5-kube-api-access-rnkbj" (OuterVolumeSpecName: "kube-api-access-rnkbj") pod "9d7493d4-c051-4d82-acf8-43f8270a5fc5" (UID: "9d7493d4-c051-4d82-acf8-43f8270a5fc5"). InnerVolumeSpecName "kube-api-access-rnkbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:57:53.488129 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.488109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c8a763-fcd3-460e-a765-f91cbb73bc6e-kube-api-access-dpd99" (OuterVolumeSpecName: "kube-api-access-dpd99") pod "04c8a763-fcd3-460e-a765-f91cbb73bc6e" (UID: "04c8a763-fcd3-460e-a765-f91cbb73bc6e"). InnerVolumeSpecName "kube-api-access-dpd99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:57:53.587484 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.587440    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpd99\" (UniqueName: \"kubernetes.io/projected/04c8a763-fcd3-460e-a765-f91cbb73bc6e-kube-api-access-dpd99\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:57:53.587484 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.587480    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7493d4-c051-4d82-acf8-43f8270a5fc5-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:57:53.587484 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.587491    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnkbj\" (UniqueName: \"kubernetes.io/projected/9d7493d4-c051-4d82-acf8-43f8270a5fc5-kube-api-access-rnkbj\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:57:53.587484 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.587499    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c8a763-fcd3-460e-a765-f91cbb73bc6e-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:57:53.884843 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.884804    2581 generic.go:356] "Generic (PLEG): container finished" podID="04c8a763-fcd3-460e-a765-f91cbb73bc6e" containerID="2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d" exitCode=0
Jul 28 23:57:53.885033 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.884853    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"
Jul 28 23:57:53.885033 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.884888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd" event={"ID":"04c8a763-fcd3-460e-a765-f91cbb73bc6e","Type":"ContainerDied","Data":"2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d"}
Jul 28 23:57:53.885033 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.884941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd" event={"ID":"04c8a763-fcd3-460e-a765-f91cbb73bc6e","Type":"ContainerDied","Data":"9fd51da0319d6c5c742ad8dd7fe2f7df3df2c1353c6f5eb9bf0ed25d84ab5173"}
Jul 28 23:57:53.885033 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.884963    2581 scope.go:117] "RemoveContainer" containerID="2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d"
Jul 28 23:57:53.886270 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.886240    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d7493d4-c051-4d82-acf8-43f8270a5fc5" containerID="76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857" exitCode=0
Jul 28 23:57:53.886354 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.886275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97" event={"ID":"9d7493d4-c051-4d82-acf8-43f8270a5fc5","Type":"ContainerDied","Data":"76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857"}
Jul 28 23:57:53.886354 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.886299    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"
Jul 28 23:57:53.886445 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.886306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97" event={"ID":"9d7493d4-c051-4d82-acf8-43f8270a5fc5","Type":"ContainerDied","Data":"4e82871010aa971187ac1edb4c5d1b099dd73217ad1c933e22fbf20d2a670743"}
Jul 28 23:57:53.901345 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.901325    2581 scope.go:117] "RemoveContainer" containerID="2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d"
Jul 28 23:57:53.901636 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:57:53.901605    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d\": container with ID starting with 2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d not found: ID does not exist" containerID="2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d"
Jul 28 23:57:53.901686 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.901651    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d"} err="failed to get container status \"2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d\": rpc error: code = NotFound desc = could not find container \"2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d\": container with ID starting with 2bc2bfc8c0bbb354a066023dd80146a808916b0e391a6113e5c4abf8965cd24d not found: ID does not exist"
Jul 28 23:57:53.901686 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.901669    2581 scope.go:117] "RemoveContainer" containerID="76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857"
Jul 28 23:57:53.910052 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.910033    2581 scope.go:117] "RemoveContainer" containerID="76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857"
Jul 28 23:57:53.910284 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:57:53.910268    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857\": container with ID starting with 76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857 not found: ID does not exist" containerID="76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857"
Jul 28 23:57:53.910336 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.910290    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857"} err="failed to get container status \"76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857\": rpc error: code = NotFound desc = could not find container \"76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857\": container with ID starting with 76010753f1440a6173b75c92326dabe059d9942c0cfe16824e97bef322837857 not found: ID does not exist"
Jul 28 23:57:53.913746 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.913723    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"]
Jul 28 23:57:53.917750 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.917731    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21449885-786c999df6-gcv97"]
Jul 28 23:57:53.927333 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.927299    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"]
Jul 28 23:57:53.929105 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:53.929086    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21449885-7f446c98b6-svwhd"]
Jul 28 23:57:54.981717 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:54.981682    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04c8a763-fcd3-460e-a765-f91cbb73bc6e" path="/var/lib/kubelet/pods/04c8a763-fcd3-460e-a765-f91cbb73bc6e/volumes"
Jul 28 23:57:54.982095 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:57:54.982060    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d7493d4-c051-4d82-acf8-43f8270a5fc5" path="/var/lib/kubelet/pods/9d7493d4-c051-4d82-acf8-43f8270a5fc5/volumes"
Jul 28 23:58:00.722482 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:00.722442    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:58:01.290809 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:01.290768    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"]
Jul 28 23:58:16.309921 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.309852    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf" podUID="da19e146-ea96-415f-869d-85a29964cd9c" containerName="main" containerID="cri-o://61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83" gracePeriod=45
Jul 28 23:58:16.563149 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.563075    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:58:16.705938 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.705906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da19e146-ea96-415f-869d-85a29964cd9c-tls-certs\") pod \"da19e146-ea96-415f-869d-85a29964cd9c\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") "
Jul 28 23:58:16.706135 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.705953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-dshm\") pod \"da19e146-ea96-415f-869d-85a29964cd9c\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") "
Jul 28 23:58:16.706135 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.705987    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krn5k\" (UniqueName: \"kubernetes.io/projected/da19e146-ea96-415f-869d-85a29964cd9c-kube-api-access-krn5k\") pod \"da19e146-ea96-415f-869d-85a29964cd9c\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") "
Jul 28 23:58:16.706135 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.706069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-home\") pod \"da19e146-ea96-415f-869d-85a29964cd9c\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") "
Jul 28 23:58:16.706135 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.706124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-tmp-dir\") pod \"da19e146-ea96-415f-869d-85a29964cd9c\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") "
Jul 28 23:58:16.706404 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.706146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-model-cache\") pod \"da19e146-ea96-415f-869d-85a29964cd9c\" (UID: \"da19e146-ea96-415f-869d-85a29964cd9c\") "
Jul 28 23:58:16.706454 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.706425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-home" (OuterVolumeSpecName: "home") pod "da19e146-ea96-415f-869d-85a29964cd9c" (UID: "da19e146-ea96-415f-869d-85a29964cd9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:58:16.706497 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.706401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da19e146-ea96-415f-869d-85a29964cd9c" (UID: "da19e146-ea96-415f-869d-85a29964cd9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:58:16.706497 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.706470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "da19e146-ea96-415f-869d-85a29964cd9c" (UID: "da19e146-ea96-415f-869d-85a29964cd9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:58:16.708329 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.708299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da19e146-ea96-415f-869d-85a29964cd9c-kube-api-access-krn5k" (OuterVolumeSpecName: "kube-api-access-krn5k") pod "da19e146-ea96-415f-869d-85a29964cd9c" (UID: "da19e146-ea96-415f-869d-85a29964cd9c"). InnerVolumeSpecName "kube-api-access-krn5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 23:58:16.708329 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.708313    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-dshm" (OuterVolumeSpecName: "dshm") pod "da19e146-ea96-415f-869d-85a29964cd9c" (UID: "da19e146-ea96-415f-869d-85a29964cd9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 23:58:16.708329 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.708317    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da19e146-ea96-415f-869d-85a29964cd9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da19e146-ea96-415f-869d-85a29964cd9c" (UID: "da19e146-ea96-415f-869d-85a29964cd9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 23:58:16.806984 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.806932    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:58:16.807178 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.807000    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:58:16.807178 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.807013    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:58:16.807178 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.807026    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da19e146-ea96-415f-869d-85a29964cd9c-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:58:16.807178 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.807040    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da19e146-ea96-415f-869d-85a29964cd9c-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:58:16.807178 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.807048    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krn5k\" (UniqueName: \"kubernetes.io/projected/da19e146-ea96-415f-869d-85a29964cd9c-kube-api-access-krn5k\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 28 23:58:16.994240 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.994208    2581 generic.go:356] "Generic (PLEG): container finished" podID="da19e146-ea96-415f-869d-85a29964cd9c" containerID="61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83" exitCode=0
Jul 28 23:58:16.994410 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.994252    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"
Jul 28 23:58:16.994410 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.994290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf" event={"ID":"da19e146-ea96-415f-869d-85a29964cd9c","Type":"ContainerDied","Data":"61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83"}
Jul 28 23:58:16.994410 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.994339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf" event={"ID":"da19e146-ea96-415f-869d-85a29964cd9c","Type":"ContainerDied","Data":"37514e88b7d09984259e06657cb1154274a8daf7fbbd5305411b4a3862bc138d"}
Jul 28 23:58:16.994410 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:16.994361    2581 scope.go:117] "RemoveContainer" containerID="61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83"
Jul 28 23:58:17.002861 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:17.002844    2581 scope.go:117] "RemoveContainer" containerID="61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83"
Jul 28 23:58:17.003140 ip-10-0-135-158 kubenswrapper[2581]: E0728 23:58:17.003121    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83\": container with ID starting with 61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83 not found: ID does not exist" containerID="61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83"
Jul 28 23:58:17.003186 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:17.003148    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83"} err="failed to get container status \"61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83\": rpc error: code = NotFound desc = could not find container \"61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83\": container with ID starting with 61c7598216094c88eb8503b0e9b1821a277ac473e79d2874e12deab022b20e83 not found: ID does not exist"
Jul 28 23:58:17.011531 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:17.011501    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"]
Jul 28 23:58:17.013097 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:17.013076    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-21449885/e2e-lifecycle-model-21449885-kserve-66545fcbbc-2z7pf"]
Jul 28 23:58:18.986642 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:58:18.986598    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da19e146-ea96-415f-869d-85a29964cd9c" path="/var/lib/kubelet/pods/da19e146-ea96-415f-869d-85a29964cd9c/volumes"
Jul 28 23:59:02.312088 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:02.312052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:59:14.723068 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:14.723026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 28 23:59:56.277209 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277171    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"]
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277532    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da19e146-ea96-415f-869d-85a29964cd9c" containerName="main"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277544    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da19e146-ea96-415f-869d-85a29964cd9c" containerName="main"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277564    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c8a763-fcd3-460e-a765-f91cbb73bc6e" containerName="payload-processing"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277570    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c8a763-fcd3-460e-a765-f91cbb73bc6e" containerName="payload-processing"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277587    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d7493d4-c051-4d82-acf8-43f8270a5fc5" containerName="payload-pre-processing"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277593    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d7493d4-c051-4d82-acf8-43f8270a5fc5" containerName="payload-pre-processing"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277675    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c8a763-fcd3-460e-a765-f91cbb73bc6e" containerName="payload-processing"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277687    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da19e146-ea96-415f-869d-85a29964cd9c" containerName="main"
Jul 28 23:59:56.277700 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.277694    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d7493d4-c051-4d82-acf8-43f8270a5fc5" containerName="payload-pre-processing"
Jul 28 23:59:56.280445 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.280430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.282927 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.282904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c418f4-dockercfg-9j8vj\""
Jul 28 23:59:56.283033 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.283013    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c418f4\""
Jul 28 23:59:56.287981 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.287956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"]
Jul 28 23:59:56.311913 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.311885    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"]
Jul 28 23:59:56.315256 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.315241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.325607 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.325580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"]
Jul 28 23:59:56.362915 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.362889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131eb3e7-96a6-456e-8b72-dd5c1104cefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.363020 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.362949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njfzs\" (UniqueName: \"kubernetes.io/projected/131eb3e7-96a6-456e-8b72-dd5c1104cefe-kube-api-access-njfzs\") pod \"payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.464213 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.464182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131eb3e7-96a6-456e-8b72-dd5c1104cefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.464381 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.464231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401b22c4-5dc7-42b3-94d7-bcdb067605ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.464381 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.464292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njfzs\" (UniqueName: \"kubernetes.io/projected/131eb3e7-96a6-456e-8b72-dd5c1104cefe-kube-api-access-njfzs\") pod \"payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.464456 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.464391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6vn\" (UniqueName: \"kubernetes.io/projected/401b22c4-5dc7-42b3-94d7-bcdb067605ce-kube-api-access-dt6vn\") pod \"payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.464913 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.464897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131eb3e7-96a6-456e-8b72-dd5c1104cefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.472587 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.472565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njfzs\" (UniqueName: \"kubernetes.io/projected/131eb3e7-96a6-456e-8b72-dd5c1104cefe-kube-api-access-njfzs\") pod \"payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.565839 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.565770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6vn\" (UniqueName: \"kubernetes.io/projected/401b22c4-5dc7-42b3-94d7-bcdb067605ce-kube-api-access-dt6vn\") pod \"payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.565839 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.565825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401b22c4-5dc7-42b3-94d7-bcdb067605ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.566449 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.566431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401b22c4-5dc7-42b3-94d7-bcdb067605ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.573726 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.573705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6vn\" (UniqueName: \"kubernetes.io/projected/401b22c4-5dc7-42b3-94d7-bcdb067605ce-kube-api-access-dt6vn\") pod \"payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.591374 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.591346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:56.625241 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.625216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:56.944013 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.943977    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"]
Jul 28 23:59:56.945435 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:59:56.945404    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131eb3e7_96a6_456e_8b72_dd5c1104cefe.slice/crio-57f540751f67adead6d86f7c0a42b2b337a6b4b4e4e3bb5379fd6139e170ff03 WatchSource:0}: Error finding container 57f540751f67adead6d86f7c0a42b2b337a6b4b4e4e3bb5379fd6139e170ff03: Status 404 returned error can't find the container with id 57f540751f67adead6d86f7c0a42b2b337a6b4b4e4e3bb5379fd6139e170ff03
Jul 28 23:59:56.970100 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:56.970037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"]
Jul 28 23:59:56.979760 ip-10-0-135-158 kubenswrapper[2581]: W0728 23:59:56.979724    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401b22c4_5dc7_42b3_94d7_bcdb067605ce.slice/crio-aa5cfb05929913509e39f395bb2833ec85f7e0b506e3ffa7abd9e4aea0d781e9 WatchSource:0}: Error finding container aa5cfb05929913509e39f395bb2833ec85f7e0b506e3ffa7abd9e4aea0d781e9: Status 404 returned error can't find the container with id aa5cfb05929913509e39f395bb2833ec85f7e0b506e3ffa7abd9e4aea0d781e9
Jul 28 23:59:57.447153 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.447094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58" event={"ID":"131eb3e7-96a6-456e-8b72-dd5c1104cefe","Type":"ContainerStarted","Data":"805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd"}
Jul 28 23:59:57.447663 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.447175    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 28 23:59:57.447663 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.447202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58" event={"ID":"131eb3e7-96a6-456e-8b72-dd5c1104cefe","Type":"ContainerStarted","Data":"57f540751f67adead6d86f7c0a42b2b337a6b4b4e4e3bb5379fd6139e170ff03"}
Jul 28 23:59:57.448796 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.448770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw" event={"ID":"401b22c4-5dc7-42b3-94d7-bcdb067605ce","Type":"ContainerStarted","Data":"ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30"}
Jul 28 23:59:57.448908 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.448819    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 28 23:59:57.448908 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.448837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw" event={"ID":"401b22c4-5dc7-42b3-94d7-bcdb067605ce","Type":"ContainerStarted","Data":"aa5cfb05929913509e39f395bb2833ec85f7e0b506e3ffa7abd9e4aea0d781e9"}
Jul 28 23:59:57.465863 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.465823    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58" podStartSLOduration=1.465809686 podStartE2EDuration="1.465809686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:59:57.463788181 +0000 UTC m=+2292.966334717" watchObservedRunningTime="2026-07-28 23:59:57.465809686 +0000 UTC m=+2292.968356217"
Jul 28 23:59:57.479157 ip-10-0-135-158 kubenswrapper[2581]: I0728 23:59:57.479103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw" podStartSLOduration=1.479089519 podStartE2EDuration="1.479089519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 23:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 23:59:57.477492192 +0000 UTC m=+2292.980038724" watchObservedRunningTime="2026-07-28 23:59:57.479089519 +0000 UTC m=+2292.981636074"
Jul 29 00:00:02.120640 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:02.120580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:00:08.456327 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:08.456294    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 29 00:00:08.456739 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:08.456387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 29 00:00:24.192838 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.192801    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"]
Jul 29 00:00:24.196138 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.196120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.198508 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.198480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b178a9-dockercfg-jdgfl\""
Jul 29 00:00:24.198645 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.198537    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b178a9\""
Jul 29 00:00:24.206016 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.205989    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"]
Jul 29 00:00:24.226580 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.226547    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"]
Jul 29 00:00:24.230018 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.229996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.239948 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.239926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"]
Jul 29 00:00:24.304551 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.304510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e724b9fe-09a0-4d79-aeb7-996693f39cf4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.304732 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.304570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96vp\" (UniqueName: \"kubernetes.io/projected/29afe255-2729-48d1-bcf1-d65da9e65f68-kube-api-access-z96vp\") pod \"payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.304732 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.304656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcz7\" (UniqueName: \"kubernetes.io/projected/e724b9fe-09a0-4d79-aeb7-996693f39cf4-kube-api-access-mkcz7\") pod \"payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.304732 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.304695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29afe255-2729-48d1-bcf1-d65da9e65f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.406000 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.405956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e724b9fe-09a0-4d79-aeb7-996693f39cf4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.406000 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.406007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96vp\" (UniqueName: \"kubernetes.io/projected/29afe255-2729-48d1-bcf1-d65da9e65f68-kube-api-access-z96vp\") pod \"payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.406181 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.406056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcz7\" (UniqueName: \"kubernetes.io/projected/e724b9fe-09a0-4d79-aeb7-996693f39cf4-kube-api-access-mkcz7\") pod \"payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.406181 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.406079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29afe255-2729-48d1-bcf1-d65da9e65f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.406842 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.406824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29afe255-2729-48d1-bcf1-d65da9e65f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.406891 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.406824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e724b9fe-09a0-4d79-aeb7-996693f39cf4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.417477 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.417449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcz7\" (UniqueName: \"kubernetes.io/projected/e724b9fe-09a0-4d79-aeb7-996693f39cf4-kube-api-access-mkcz7\") pod \"payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.417477 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.417462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96vp\" (UniqueName: \"kubernetes.io/projected/29afe255-2729-48d1-bcf1-d65da9e65f68-kube-api-access-z96vp\") pod \"payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.523574 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.523483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:24.541907 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.541858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:24.879803 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.879775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"]
Jul 29 00:00:24.881381 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:00:24.881351    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29afe255_2729_48d1_bcf1_d65da9e65f68.slice/crio-a449849866939f9f22f6d97b8e9a1e8a6e7a3af4427fee1613a284e9f53d92fe WatchSource:0}: Error finding container a449849866939f9f22f6d97b8e9a1e8a6e7a3af4427fee1613a284e9f53d92fe: Status 404 returned error can't find the container with id a449849866939f9f22f6d97b8e9a1e8a6e7a3af4427fee1613a284e9f53d92fe
Jul 29 00:00:24.908558 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:24.908536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"]
Jul 29 00:00:25.581908 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.581870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j" event={"ID":"29afe255-2729-48d1-bcf1-d65da9e65f68","Type":"ContainerStarted","Data":"17756c7018cd79003b7f7237491bd098baf913d743e50624070fdd33c0839427"}
Jul 29 00:00:25.582377 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.581937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j" event={"ID":"29afe255-2729-48d1-bcf1-d65da9e65f68","Type":"ContainerStarted","Data":"a449849866939f9f22f6d97b8e9a1e8a6e7a3af4427fee1613a284e9f53d92fe"}
Jul 29 00:00:25.582377 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.581968    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:00:25.583459 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.583428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh" event={"ID":"e724b9fe-09a0-4d79-aeb7-996693f39cf4","Type":"ContainerStarted","Data":"b3474d6b2311470b4166d0726df9c7bb299b60b2ddd1c0ee2064ff8dc388aa05"}
Jul 29 00:00:25.583578 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.583481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh" event={"ID":"e724b9fe-09a0-4d79-aeb7-996693f39cf4","Type":"ContainerStarted","Data":"4f4e3aa0ae10f73d763cc2afd960b8ad2bd182abaa6293862fed914bd71a3112"}
Jul 29 00:00:25.583578 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.583525    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:25.596674 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.596600    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j" podStartSLOduration=1.59658433 podStartE2EDuration="1.59658433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:00:25.59634781 +0000 UTC m=+2321.098894343" watchObservedRunningTime="2026-07-29 00:00:25.59658433 +0000 UTC m=+2321.099130862"
Jul 29 00:00:25.610421 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:25.610368    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh" podStartSLOduration=1.610350304 podStartE2EDuration="1.610350304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:00:25.610287161 +0000 UTC m=+2321.112833692" watchObservedRunningTime="2026-07-29 00:00:25.610350304 +0000 UTC m=+2321.112896836"
Jul 29 00:00:31.917863 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:31.917827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:00:36.591374 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:36.591347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:00:36.591882 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:00:36.591421    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:01:03.014993 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.014903    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"]
Jul 29 00:01:03.018882 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.018860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.022578 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.022552    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c418f4\"/\"kube-root-ca.crt\""
Jul 29 00:01:03.022728 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.022676    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c418f4\"/\"default-dockercfg-q5d77\""
Jul 29 00:01:03.022728 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.022671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c418f4\"/\"test-model-c418f4-kserve-self-signed-certs\""
Jul 29 00:01:03.022842 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.022722    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c418f4\"/\"openshift-service-ca.crt\""
Jul 29 00:01:03.029254 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.029230    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"]
Jul 29 00:01:03.060404 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.060373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-dshm\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.060574 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.060415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tls-certs\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.060574 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.060438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-model-cache\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.060574 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.060551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stl9v\" (UniqueName: \"kubernetes.io/projected/6ae4f159-3e80-41af-83c3-d784c3b71c3d-kube-api-access-stl9v\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.060703 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.060614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-home\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.060703 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.060658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tmp-dir\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162185 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stl9v\" (UniqueName: \"kubernetes.io/projected/6ae4f159-3e80-41af-83c3-d784c3b71c3d-kube-api-access-stl9v\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162404 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-home\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162404 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tmp-dir\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162404 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-dshm\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162404 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tls-certs\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162404 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-model-cache\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162699 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-home\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162699 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tmp-dir\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.162830 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.162809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-model-cache\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.164765 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.164743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-dshm\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.165031 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.165015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tls-certs\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.169959 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.169932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stl9v\" (UniqueName: \"kubernetes.io/projected/6ae4f159-3e80-41af-83c3-d784c3b71c3d-kube-api-access-stl9v\") pod \"test-model-c418f4-kserve-7755ddb797-w7l9n\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") " pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.329130 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.329024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:03.462178 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.462154    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"]
Jul 29 00:01:03.463991 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:01:03.463963    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ae4f159_3e80_41af_83c3_d784c3b71c3d.slice/crio-732484e6442e548e948c1c17579974230a6bd23953db12467a69fe251caa5080 WatchSource:0}: Error finding container 732484e6442e548e948c1c17579974230a6bd23953db12467a69fe251caa5080: Status 404 returned error can't find the container with id 732484e6442e548e948c1c17579974230a6bd23953db12467a69fe251caa5080
Jul 29 00:01:03.762349 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.762311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n" event={"ID":"6ae4f159-3e80-41af-83c3-d784c3b71c3d","Type":"ContainerStarted","Data":"95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307"}
Jul 29 00:01:03.762349 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.762354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n" event={"ID":"6ae4f159-3e80-41af-83c3-d784c3b71c3d","Type":"ContainerStarted","Data":"732484e6442e548e948c1c17579974230a6bd23953db12467a69fe251caa5080"}
Jul 29 00:01:03.783648 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:03.783579    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n" podStartSLOduration=1.783563736 podStartE2EDuration="1.783563736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:01:03.781767178 +0000 UTC m=+2359.284313709" watchObservedRunningTime="2026-07-29 00:01:03.783563736 +0000 UTC m=+2359.286110267"
Jul 29 00:01:07.418749 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:07.418718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:01:13.330056 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:13.330025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:13.330515 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:13.330292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:13.342761 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:13.342738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:13.821573 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:13.821544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:01:33.645151 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.645110    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"]
Jul 29 00:01:33.648889 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.648867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.653952 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.653930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b178a9\"/\"test-model-b178a9-kserve-self-signed-certs\""
Jul 29 00:01:33.654061 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.653968    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b178a9\"/\"kube-root-ca.crt\""
Jul 29 00:01:33.657266 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.657250    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b178a9\"/\"default-dockercfg-ccrpp\""
Jul 29 00:01:33.657547 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.657531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b178a9\"/\"openshift-service-ca.crt\""
Jul 29 00:01:33.670288 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.670263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"]
Jul 29 00:01:33.749388 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.749352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-home\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.749388 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.749389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-tmp-dir\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.749605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.749495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7bv\" (UniqueName: \"kubernetes.io/projected/a5358049-4467-4901-a849-deb6de2bb479-kube-api-access-xq7bv\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.749605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.749515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5358049-4467-4901-a849-deb6de2bb479-tls-certs\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.749605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.749532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-dshm\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.749605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.749553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-model-cache\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.850938 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.850901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7bv\" (UniqueName: \"kubernetes.io/projected/a5358049-4467-4901-a849-deb6de2bb479-kube-api-access-xq7bv\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851114 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.850943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5358049-4467-4901-a849-deb6de2bb479-tls-certs\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851114 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.850965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-dshm\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851114 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.850986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-model-cache\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851289 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.851106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-home\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851289 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.851149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-tmp-dir\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851401 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.851385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-model-cache\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851474 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.851456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-home\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.851513 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.851490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-tmp-dir\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.853275 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.853252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-dshm\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.853613 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.853595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5358049-4467-4901-a849-deb6de2bb479-tls-certs\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.860781 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.860759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7bv\" (UniqueName: \"kubernetes.io/projected/a5358049-4467-4901-a849-deb6de2bb479-kube-api-access-xq7bv\") pod \"test-model-b178a9-kserve-555cf4c877-hnrl5\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") " pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:33.958674 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:33.958550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:34.095982 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:34.095957    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"]
Jul 29 00:01:34.098126 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:01:34.098098    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5358049_4467_4901_a849_deb6de2bb479.slice/crio-2c42393e8d3228375f95ea0927c6938a2d100bea3029b48f4c906735ec81a986 WatchSource:0}: Error finding container 2c42393e8d3228375f95ea0927c6938a2d100bea3029b48f4c906735ec81a986: Status 404 returned error can't find the container with id 2c42393e8d3228375f95ea0927c6938a2d100bea3029b48f4c906735ec81a986
Jul 29 00:01:34.911270 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:34.911231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5" event={"ID":"a5358049-4467-4901-a849-deb6de2bb479","Type":"ContainerStarted","Data":"17a34e36b626cf7f20ba869299e7662cf158a574e4f1254d0a57ccbf480cf80a"}
Jul 29 00:01:34.911762 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:34.911279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5" event={"ID":"a5358049-4467-4901-a849-deb6de2bb479","Type":"ContainerStarted","Data":"2c42393e8d3228375f95ea0927c6938a2d100bea3029b48f4c906735ec81a986"}
Jul 29 00:01:34.929016 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:34.928968    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5" podStartSLOduration=1.928951337 podStartE2EDuration="1.928951337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:01:34.927789298 +0000 UTC m=+2390.430335832" watchObservedRunningTime="2026-07-29 00:01:34.928951337 +0000 UTC m=+2390.431497867"
Jul 29 00:01:37.523317 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:37.523280    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:01:43.958902 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:43.958864    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:43.959361 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:43.958938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:43.971523 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:43.971492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:44.969839 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:44.969809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:01:51.812132 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:01:51.812095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:02:13.123903 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:13.123815    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"]
Jul 29 00:02:14.033537 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:14.033505    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"]
Jul 29 00:02:17.716277 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:17.716236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:02:24.645136 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.645101    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"]
Jul 29 00:02:24.645614 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.645323    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw" podUID="401b22c4-5dc7-42b3-94d7-bcdb067605ce" containerName="payload-processing" containerID="cri-o://ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30" gracePeriod=30
Jul 29 00:02:24.653829 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.653791    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"]
Jul 29 00:02:24.654037 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.653993    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58" podUID="131eb3e7-96a6-456e-8b72-dd5c1104cefe" containerName="payload-pre-processing" containerID="cri-o://805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd" gracePeriod=30
Jul 29 00:02:24.906933 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.906872    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 29 00:02:24.929843 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.929814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt6vn\" (UniqueName: \"kubernetes.io/projected/401b22c4-5dc7-42b3-94d7-bcdb067605ce-kube-api-access-dt6vn\") pod \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") "
Jul 29 00:02:24.930017 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.929999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401b22c4-5dc7-42b3-94d7-bcdb067605ce-plugins-config-volume\") pod \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\" (UID: \"401b22c4-5dc7-42b3-94d7-bcdb067605ce\") "
Jul 29 00:02:24.930424 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.930391    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401b22c4-5dc7-42b3-94d7-bcdb067605ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "401b22c4-5dc7-42b3-94d7-bcdb067605ce" (UID: "401b22c4-5dc7-42b3-94d7-bcdb067605ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:02:24.932579 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.932549    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401b22c4-5dc7-42b3-94d7-bcdb067605ce-kube-api-access-dt6vn" (OuterVolumeSpecName: "kube-api-access-dt6vn") pod "401b22c4-5dc7-42b3-94d7-bcdb067605ce" (UID: "401b22c4-5dc7-42b3-94d7-bcdb067605ce"). InnerVolumeSpecName "kube-api-access-dt6vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:02:24.935276 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:24.935254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 29 00:02:25.031007 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.030982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131eb3e7-96a6-456e-8b72-dd5c1104cefe-plugins-config-volume\") pod \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") "
Jul 29 00:02:25.031169 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.031036    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njfzs\" (UniqueName: \"kubernetes.io/projected/131eb3e7-96a6-456e-8b72-dd5c1104cefe-kube-api-access-njfzs\") pod \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\" (UID: \"131eb3e7-96a6-456e-8b72-dd5c1104cefe\") "
Jul 29 00:02:25.031293 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.031277    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401b22c4-5dc7-42b3-94d7-bcdb067605ce-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:25.031364 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.031294    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt6vn\" (UniqueName: \"kubernetes.io/projected/401b22c4-5dc7-42b3-94d7-bcdb067605ce-kube-api-access-dt6vn\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:25.031364 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.031341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131eb3e7-96a6-456e-8b72-dd5c1104cefe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "131eb3e7-96a6-456e-8b72-dd5c1104cefe" (UID: "131eb3e7-96a6-456e-8b72-dd5c1104cefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:02:25.033148 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.033126    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131eb3e7-96a6-456e-8b72-dd5c1104cefe-kube-api-access-njfzs" (OuterVolumeSpecName: "kube-api-access-njfzs") pod "131eb3e7-96a6-456e-8b72-dd5c1104cefe" (UID: "131eb3e7-96a6-456e-8b72-dd5c1104cefe"). InnerVolumeSpecName "kube-api-access-njfzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:02:25.132492 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.132456    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131eb3e7-96a6-456e-8b72-dd5c1104cefe-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:25.132492 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.132486    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njfzs\" (UniqueName: \"kubernetes.io/projected/131eb3e7-96a6-456e-8b72-dd5c1104cefe-kube-api-access-njfzs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:25.157730 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.157621    2581 generic.go:356] "Generic (PLEG): container finished" podID="401b22c4-5dc7-42b3-94d7-bcdb067605ce" containerID="ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30" exitCode=0
Jul 29 00:02:25.157730 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.157700    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"
Jul 29 00:02:25.157730 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.157712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw" event={"ID":"401b22c4-5dc7-42b3-94d7-bcdb067605ce","Type":"ContainerDied","Data":"ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30"}
Jul 29 00:02:25.157985 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.157760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw" event={"ID":"401b22c4-5dc7-42b3-94d7-bcdb067605ce","Type":"ContainerDied","Data":"aa5cfb05929913509e39f395bb2833ec85f7e0b506e3ffa7abd9e4aea0d781e9"}
Jul 29 00:02:25.157985 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.157786    2581 scope.go:117] "RemoveContainer" containerID="ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30"
Jul 29 00:02:25.159340 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.159054    2581 generic.go:356] "Generic (PLEG): container finished" podID="131eb3e7-96a6-456e-8b72-dd5c1104cefe" containerID="805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd" exitCode=0
Jul 29 00:02:25.159340 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.159084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58" event={"ID":"131eb3e7-96a6-456e-8b72-dd5c1104cefe","Type":"ContainerDied","Data":"805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd"}
Jul 29 00:02:25.159340 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.159107    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"
Jul 29 00:02:25.159340 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.159157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58" event={"ID":"131eb3e7-96a6-456e-8b72-dd5c1104cefe","Type":"ContainerDied","Data":"57f540751f67adead6d86f7c0a42b2b337a6b4b4e4e3bb5379fd6139e170ff03"}
Jul 29 00:02:25.166913 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.166893    2581 scope.go:117] "RemoveContainer" containerID="ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30"
Jul 29 00:02:25.167321 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:02:25.167175    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30\": container with ID starting with ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30 not found: ID does not exist" containerID="ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30"
Jul 29 00:02:25.167321 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.167205    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30"} err="failed to get container status \"ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30\": rpc error: code = NotFound desc = could not find container \"ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30\": container with ID starting with ed0f7d05209c61192a204e3a0309a763cf7adcfeeed03f865596c66da83beb30 not found: ID does not exist"
Jul 29 00:02:25.167321 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.167222    2581 scope.go:117] "RemoveContainer" containerID="805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd"
Jul 29 00:02:25.175554 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.175533    2581 scope.go:117] "RemoveContainer" containerID="805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd"
Jul 29 00:02:25.175838 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:02:25.175819    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd\": container with ID starting with 805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd not found: ID does not exist" containerID="805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd"
Jul 29 00:02:25.175914 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.175848    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd"} err="failed to get container status \"805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd\": rpc error: code = NotFound desc = could not find container \"805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd\": container with ID starting with 805eda5c9b0ec053531d6ede2d06a51c159d839791015fe56ae5460884d04fbd not found: ID does not exist"
Jul 29 00:02:25.176366 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.176348    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"]
Jul 29 00:02:25.180334 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.180314    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c418f4-564d64ddf4-m4bnw"]
Jul 29 00:02:25.191591 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.191570    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"]
Jul 29 00:02:25.196072 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:25.196052    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c418f4-6df5d56f84-2bx58"]
Jul 29 00:02:26.981583 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:26.981553    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="131eb3e7-96a6-456e-8b72-dd5c1104cefe" path="/var/lib/kubelet/pods/131eb3e7-96a6-456e-8b72-dd5c1104cefe/volumes"
Jul 29 00:02:26.981976 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:26.981936    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401b22c4-5dc7-42b3-94d7-bcdb067605ce" path="/var/lib/kubelet/pods/401b22c4-5dc7-42b3-94d7-bcdb067605ce/volumes"
Jul 29 00:02:28.143622 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.143583    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n" podUID="6ae4f159-3e80-41af-83c3-d784c3b71c3d" containerName="main" containerID="cri-o://95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307" gracePeriod=45
Jul 29 00:02:28.384673 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.384651    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:02:28.464946 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.464864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-model-cache\") pod \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") "
Jul 29 00:02:28.464946 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.464916    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-home\") pod \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") "
Jul 29 00:02:28.465140 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.464959    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tls-certs\") pod \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") "
Jul 29 00:02:28.465140 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.464986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-dshm\") pod \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") "
Jul 29 00:02:28.465140 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.465004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stl9v\" (UniqueName: \"kubernetes.io/projected/6ae4f159-3e80-41af-83c3-d784c3b71c3d-kube-api-access-stl9v\") pod \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") "
Jul 29 00:02:28.465140 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.465026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tmp-dir\") pod \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\" (UID: \"6ae4f159-3e80-41af-83c3-d784c3b71c3d\") "
Jul 29 00:02:28.465340 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.465159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ae4f159-3e80-41af-83c3-d784c3b71c3d" (UID: "6ae4f159-3e80-41af-83c3-d784c3b71c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:28.465340 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.465332    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:28.465452 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.465338    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-home" (OuterVolumeSpecName: "home") pod "6ae4f159-3e80-41af-83c3-d784c3b71c3d" (UID: "6ae4f159-3e80-41af-83c3-d784c3b71c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:28.465452 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.465381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ae4f159-3e80-41af-83c3-d784c3b71c3d" (UID: "6ae4f159-3e80-41af-83c3-d784c3b71c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:28.467259 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.467238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-dshm" (OuterVolumeSpecName: "dshm") pod "6ae4f159-3e80-41af-83c3-d784c3b71c3d" (UID: "6ae4f159-3e80-41af-83c3-d784c3b71c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:28.467334 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.467284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae4f159-3e80-41af-83c3-d784c3b71c3d-kube-api-access-stl9v" (OuterVolumeSpecName: "kube-api-access-stl9v") pod "6ae4f159-3e80-41af-83c3-d784c3b71c3d" (UID: "6ae4f159-3e80-41af-83c3-d784c3b71c3d"). InnerVolumeSpecName "kube-api-access-stl9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:02:28.467392 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.467344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ae4f159-3e80-41af-83c3-d784c3b71c3d" (UID: "6ae4f159-3e80-41af-83c3-d784c3b71c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:02:28.566469 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.566435    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:28.566469 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.566462    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:28.566469 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.566473    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stl9v\" (UniqueName: \"kubernetes.io/projected/6ae4f159-3e80-41af-83c3-d784c3b71c3d-kube-api-access-stl9v\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:28.566737 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.566483    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:28.566737 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:28.566494    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae4f159-3e80-41af-83c3-d784c3b71c3d-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:29.053486 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.053448    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5" podUID="a5358049-4467-4901-a849-deb6de2bb479" containerName="main" containerID="cri-o://17a34e36b626cf7f20ba869299e7662cf158a574e4f1254d0a57ccbf480cf80a" gracePeriod=45
Jul 29 00:02:29.183239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.183210    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ae4f159-3e80-41af-83c3-d784c3b71c3d" containerID="95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307" exitCode=0
Jul 29 00:02:29.183642 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.183255    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"
Jul 29 00:02:29.183642 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.183289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n" event={"ID":"6ae4f159-3e80-41af-83c3-d784c3b71c3d","Type":"ContainerDied","Data":"95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307"}
Jul 29 00:02:29.183642 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.183341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n" event={"ID":"6ae4f159-3e80-41af-83c3-d784c3b71c3d","Type":"ContainerDied","Data":"732484e6442e548e948c1c17579974230a6bd23953db12467a69fe251caa5080"}
Jul 29 00:02:29.183642 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.183369    2581 scope.go:117] "RemoveContainer" containerID="95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307"
Jul 29 00:02:29.185239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.185214    2581 generic.go:356] "Generic (PLEG): container finished" podID="a5358049-4467-4901-a849-deb6de2bb479" containerID="17a34e36b626cf7f20ba869299e7662cf158a574e4f1254d0a57ccbf480cf80a" exitCode=0
Jul 29 00:02:29.185357 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.185292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5" event={"ID":"a5358049-4467-4901-a849-deb6de2bb479","Type":"ContainerDied","Data":"17a34e36b626cf7f20ba869299e7662cf158a574e4f1254d0a57ccbf480cf80a"}
Jul 29 00:02:29.192663 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.192641    2581 scope.go:117] "RemoveContainer" containerID="95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307"
Jul 29 00:02:29.192916 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:02:29.192896    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307\": container with ID starting with 95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307 not found: ID does not exist" containerID="95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307"
Jul 29 00:02:29.192981 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.192924    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307"} err="failed to get container status \"95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307\": rpc error: code = NotFound desc = could not find container \"95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307\": container with ID starting with 95a4fe848b860e74466ec66ca90f4acc62246b4aa02bc1bbdd14d07805129307 not found: ID does not exist"
Jul 29 00:02:29.198816 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.198795    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"]
Jul 29 00:02:29.202664 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.202613    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c418f4/test-model-c418f4-kserve-7755ddb797-w7l9n"]
Jul 29 00:02:29.317638 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.317597    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:02:29.374519 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374490    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-model-cache\") pod \"a5358049-4467-4901-a849-deb6de2bb479\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") "
Jul 29 00:02:29.374519 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq7bv\" (UniqueName: \"kubernetes.io/projected/a5358049-4467-4901-a849-deb6de2bb479-kube-api-access-xq7bv\") pod \"a5358049-4467-4901-a849-deb6de2bb479\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") "
Jul 29 00:02:29.374720 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-dshm\") pod \"a5358049-4467-4901-a849-deb6de2bb479\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") "
Jul 29 00:02:29.374720 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374618    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-home\") pod \"a5358049-4467-4901-a849-deb6de2bb479\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") "
Jul 29 00:02:29.374720 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5358049-4467-4901-a849-deb6de2bb479-tls-certs\") pod \"a5358049-4467-4901-a849-deb6de2bb479\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") "
Jul 29 00:02:29.374831 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374735    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-tmp-dir\") pod \"a5358049-4467-4901-a849-deb6de2bb479\" (UID: \"a5358049-4467-4901-a849-deb6de2bb479\") "
Jul 29 00:02:29.374831 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5358049-4467-4901-a849-deb6de2bb479" (UID: "a5358049-4467-4901-a849-deb6de2bb479"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:29.374939 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.374913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-home" (OuterVolumeSpecName: "home") pod "a5358049-4467-4901-a849-deb6de2bb479" (UID: "a5358049-4467-4901-a849-deb6de2bb479"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:29.375066 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.375043    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:29.375066 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.375058    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5358049-4467-4901-a849-deb6de2bb479" (UID: "a5358049-4467-4901-a849-deb6de2bb479"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:29.375247 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.375069    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:29.376857 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.376827    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5358049-4467-4901-a849-deb6de2bb479-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5358049-4467-4901-a849-deb6de2bb479" (UID: "a5358049-4467-4901-a849-deb6de2bb479"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:02:29.376954 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.376876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-dshm" (OuterVolumeSpecName: "dshm") pod "a5358049-4467-4901-a849-deb6de2bb479" (UID: "a5358049-4467-4901-a849-deb6de2bb479"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:02:29.376954 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.376883    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5358049-4467-4901-a849-deb6de2bb479-kube-api-access-xq7bv" (OuterVolumeSpecName: "kube-api-access-xq7bv") pod "a5358049-4467-4901-a849-deb6de2bb479" (UID: "a5358049-4467-4901-a849-deb6de2bb479"). InnerVolumeSpecName "kube-api-access-xq7bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:02:29.475994 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.475965    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5358049-4467-4901-a849-deb6de2bb479-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:29.475994 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.475987    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:29.475994 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.475996    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq7bv\" (UniqueName: \"kubernetes.io/projected/a5358049-4467-4901-a849-deb6de2bb479-kube-api-access-xq7bv\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:29.476213 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:29.476006    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5358049-4467-4901-a849-deb6de2bb479-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:30.191274 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:30.191241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"
Jul 29 00:02:30.191768 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:30.191295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5" event={"ID":"a5358049-4467-4901-a849-deb6de2bb479","Type":"ContainerDied","Data":"2c42393e8d3228375f95ea0927c6938a2d100bea3029b48f4c906735ec81a986"}
Jul 29 00:02:30.191768 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:30.191351    2581 scope.go:117] "RemoveContainer" containerID="17a34e36b626cf7f20ba869299e7662cf158a574e4f1254d0a57ccbf480cf80a"
Jul 29 00:02:30.213909 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:30.213885    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"]
Jul 29 00:02:30.218354 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:30.218331    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b178a9/test-model-b178a9-kserve-555cf4c877-hnrl5"]
Jul 29 00:02:30.982050 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:30.982020    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae4f159-3e80-41af-83c3-d784c3b71c3d" path="/var/lib/kubelet/pods/6ae4f159-3e80-41af-83c3-d784c3b71c3d/volumes"
Jul 29 00:02:30.982429 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:30.982415    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5358049-4467-4901-a849-deb6de2bb479" path="/var/lib/kubelet/pods/a5358049-4467-4901-a849-deb6de2bb479/volumes"
Jul 29 00:02:33.617332 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:33.617289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:02:42.039394 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.039363    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"]
Jul 29 00:02:42.039857 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.039655    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh" podUID="e724b9fe-09a0-4d79-aeb7-996693f39cf4" containerName="payload-processing" containerID="cri-o://b3474d6b2311470b4166d0726df9c7bb299b60b2ddd1c0ee2064ff8dc388aa05" gracePeriod=30
Jul 29 00:02:42.048035 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.048004    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"]
Jul 29 00:02:42.048340 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.048235    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j" podUID="29afe255-2729-48d1-bcf1-d65da9e65f68" containerName="payload-pre-processing" containerID="cri-o://17756c7018cd79003b7f7237491bd098baf913d743e50624070fdd33c0839427" gracePeriod=30
Jul 29 00:02:42.257432 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.257231    2581 generic.go:356] "Generic (PLEG): container finished" podID="29afe255-2729-48d1-bcf1-d65da9e65f68" containerID="17756c7018cd79003b7f7237491bd098baf913d743e50624070fdd33c0839427" exitCode=0
Jul 29 00:02:42.257580 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.257388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j" event={"ID":"29afe255-2729-48d1-bcf1-d65da9e65f68","Type":"ContainerDied","Data":"17756c7018cd79003b7f7237491bd098baf913d743e50624070fdd33c0839427"}
Jul 29 00:02:42.259342 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.259307    2581 generic.go:356] "Generic (PLEG): container finished" podID="e724b9fe-09a0-4d79-aeb7-996693f39cf4" containerID="b3474d6b2311470b4166d0726df9c7bb299b60b2ddd1c0ee2064ff8dc388aa05" exitCode=0
Jul 29 00:02:42.259473 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.259386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh" event={"ID":"e724b9fe-09a0-4d79-aeb7-996693f39cf4","Type":"ContainerDied","Data":"b3474d6b2311470b4166d0726df9c7bb299b60b2ddd1c0ee2064ff8dc388aa05"}
Jul 29 00:02:42.344197 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.344174    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:02:42.347026 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.347007    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:02:42.497119 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.497088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e724b9fe-09a0-4d79-aeb7-996693f39cf4-plugins-config-volume\") pod \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") "
Jul 29 00:02:42.497119 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.497126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkcz7\" (UniqueName: \"kubernetes.io/projected/e724b9fe-09a0-4d79-aeb7-996693f39cf4-kube-api-access-mkcz7\") pod \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\" (UID: \"e724b9fe-09a0-4d79-aeb7-996693f39cf4\") "
Jul 29 00:02:42.497382 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.497162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29afe255-2729-48d1-bcf1-d65da9e65f68-plugins-config-volume\") pod \"29afe255-2729-48d1-bcf1-d65da9e65f68\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") "
Jul 29 00:02:42.497382 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.497209    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z96vp\" (UniqueName: \"kubernetes.io/projected/29afe255-2729-48d1-bcf1-d65da9e65f68-kube-api-access-z96vp\") pod \"29afe255-2729-48d1-bcf1-d65da9e65f68\" (UID: \"29afe255-2729-48d1-bcf1-d65da9e65f68\") "
Jul 29 00:02:42.497591 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.497566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e724b9fe-09a0-4d79-aeb7-996693f39cf4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e724b9fe-09a0-4d79-aeb7-996693f39cf4" (UID: "e724b9fe-09a0-4d79-aeb7-996693f39cf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:02:42.497698 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.497585    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29afe255-2729-48d1-bcf1-d65da9e65f68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29afe255-2729-48d1-bcf1-d65da9e65f68" (UID: "29afe255-2729-48d1-bcf1-d65da9e65f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:02:42.499294 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.499275    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e724b9fe-09a0-4d79-aeb7-996693f39cf4-kube-api-access-mkcz7" (OuterVolumeSpecName: "kube-api-access-mkcz7") pod "e724b9fe-09a0-4d79-aeb7-996693f39cf4" (UID: "e724b9fe-09a0-4d79-aeb7-996693f39cf4"). InnerVolumeSpecName "kube-api-access-mkcz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:02:42.499431 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.499416    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29afe255-2729-48d1-bcf1-d65da9e65f68-kube-api-access-z96vp" (OuterVolumeSpecName: "kube-api-access-z96vp") pod "29afe255-2729-48d1-bcf1-d65da9e65f68" (UID: "29afe255-2729-48d1-bcf1-d65da9e65f68"). InnerVolumeSpecName "kube-api-access-z96vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:02:42.598771 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.598678    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z96vp\" (UniqueName: \"kubernetes.io/projected/29afe255-2729-48d1-bcf1-d65da9e65f68-kube-api-access-z96vp\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:42.598771 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.598715    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e724b9fe-09a0-4d79-aeb7-996693f39cf4-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:42.598771 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.598725    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkcz7\" (UniqueName: \"kubernetes.io/projected/e724b9fe-09a0-4d79-aeb7-996693f39cf4-kube-api-access-mkcz7\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:42.598771 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:42.598733    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29afe255-2729-48d1-bcf1-d65da9e65f68-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:02:43.265395 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.265304    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"
Jul 29 00:02:43.265395 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.265338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh" event={"ID":"e724b9fe-09a0-4d79-aeb7-996693f39cf4","Type":"ContainerDied","Data":"4f4e3aa0ae10f73d763cc2afd960b8ad2bd182abaa6293862fed914bd71a3112"}
Jul 29 00:02:43.265926 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.265407    2581 scope.go:117] "RemoveContainer" containerID="b3474d6b2311470b4166d0726df9c7bb299b60b2ddd1c0ee2064ff8dc388aa05"
Jul 29 00:02:43.266768 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.266748    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"
Jul 29 00:02:43.266842 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.266771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j" event={"ID":"29afe255-2729-48d1-bcf1-d65da9e65f68","Type":"ContainerDied","Data":"a449849866939f9f22f6d97b8e9a1e8a6e7a3af4427fee1613a284e9f53d92fe"}
Jul 29 00:02:43.276127 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.276072    2581 scope.go:117] "RemoveContainer" containerID="17756c7018cd79003b7f7237491bd098baf913d743e50624070fdd33c0839427"
Jul 29 00:02:43.281963 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.281940    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"]
Jul 29 00:02:43.285836 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.285815    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b178a9-c6c47dbc6-q9sxh"]
Jul 29 00:02:43.298613 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.298590    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"]
Jul 29 00:02:43.301941 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:43.301918    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b178a9-754f88fb4d-npk7j"]
Jul 29 00:02:44.981502 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:44.981471    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29afe255-2729-48d1-bcf1-d65da9e65f68" path="/var/lib/kubelet/pods/29afe255-2729-48d1-bcf1-d65da9e65f68/volumes"
Jul 29 00:02:44.981875 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:44.981860    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e724b9fe-09a0-4d79-aeb7-996693f39cf4" path="/var/lib/kubelet/pods/e724b9fe-09a0-4d79-aeb7-996693f39cf4/volumes"
Jul 29 00:02:45.032407 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:45.032371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:02:53.403616 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.403580    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"]
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.403949    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29afe255-2729-48d1-bcf1-d65da9e65f68" containerName="payload-pre-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.403961    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29afe255-2729-48d1-bcf1-d65da9e65f68" containerName="payload-pre-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.403971    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131eb3e7-96a6-456e-8b72-dd5c1104cefe" containerName="payload-pre-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.403977    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="131eb3e7-96a6-456e-8b72-dd5c1104cefe" containerName="payload-pre-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.403988    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae4f159-3e80-41af-83c3-d784c3b71c3d" containerName="main"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.403993    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae4f159-3e80-41af-83c3-d784c3b71c3d" containerName="main"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404000    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5358049-4467-4901-a849-deb6de2bb479" containerName="main"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404005    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5358049-4467-4901-a849-deb6de2bb479" containerName="main"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404011    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e724b9fe-09a0-4d79-aeb7-996693f39cf4" containerName="payload-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404016    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e724b9fe-09a0-4d79-aeb7-996693f39cf4" containerName="payload-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404022    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401b22c4-5dc7-42b3-94d7-bcdb067605ce" containerName="payload-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404027    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="401b22c4-5dc7-42b3-94d7-bcdb067605ce" containerName="payload-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404079    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5358049-4467-4901-a849-deb6de2bb479" containerName="main"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404087    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae4f159-3e80-41af-83c3-d784c3b71c3d" containerName="main"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404094    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="29afe255-2729-48d1-bcf1-d65da9e65f68" containerName="payload-pre-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404100    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="401b22c4-5dc7-42b3-94d7-bcdb067605ce" containerName="payload-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404109    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="131eb3e7-96a6-456e-8b72-dd5c1104cefe" containerName="payload-pre-processing"
Jul 29 00:02:53.404125 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.404114    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e724b9fe-09a0-4d79-aeb7-996693f39cf4" containerName="payload-processing"
Jul 29 00:02:53.407050 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.407034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.411004 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.410970    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-519e1e-dockercfg-7z9rw\""
Jul 29 00:02:53.411182 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.411163    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-519e1e\""
Jul 29 00:02:53.417652 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.417615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"]
Jul 29 00:02:53.440177 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.440145    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"]
Jul 29 00:02:53.443586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.443568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.453527 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.453499    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"]
Jul 29 00:02:53.499114 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.499076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwmq\" (UniqueName: \"kubernetes.io/projected/89c6c2d3-9727-4933-8f17-4fe98124ebaf-kube-api-access-xmwmq\") pod \"payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.499299 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.499123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c6c2d3-9727-4933-8f17-4fe98124ebaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.499299 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.499145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b344c2b-d638-4e5c-8218-5fabe86f512f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") " pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.499299 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.499232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwrjj\" (UniqueName: \"kubernetes.io/projected/6b344c2b-d638-4e5c-8218-5fabe86f512f-kube-api-access-gwrjj\") pod \"payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") " pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.600471 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.600427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwmq\" (UniqueName: \"kubernetes.io/projected/89c6c2d3-9727-4933-8f17-4fe98124ebaf-kube-api-access-xmwmq\") pod \"payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.600471 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.600477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c6c2d3-9727-4933-8f17-4fe98124ebaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.600717 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.600499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b344c2b-d638-4e5c-8218-5fabe86f512f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") " pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.600717 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.600552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwrjj\" (UniqueName: \"kubernetes.io/projected/6b344c2b-d638-4e5c-8218-5fabe86f512f-kube-api-access-gwrjj\") pod \"payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") " pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.601247 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.601225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b344c2b-d638-4e5c-8218-5fabe86f512f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") " pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.601290 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.601245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c6c2d3-9727-4933-8f17-4fe98124ebaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.608652 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.608595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwmq\" (UniqueName: \"kubernetes.io/projected/89c6c2d3-9727-4933-8f17-4fe98124ebaf-kube-api-access-xmwmq\") pod \"payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.608803 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.608723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwrjj\" (UniqueName: \"kubernetes.io/projected/6b344c2b-d638-4e5c-8218-5fabe86f512f-kube-api-access-gwrjj\") pod \"payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") " pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.717935 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.717850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:53.754032 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.753998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:53.864041 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.864010    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"]
Jul 29 00:02:53.866209 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:02:53.866181    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c6c2d3_9727_4933_8f17_4fe98124ebaf.slice/crio-b1875c97037b11c3ec9efbbcb058959f6418b0da5365ccf21e06ba026762d8ed WatchSource:0}: Error finding container b1875c97037b11c3ec9efbbcb058959f6418b0da5365ccf21e06ba026762d8ed: Status 404 returned error can't find the container with id b1875c97037b11c3ec9efbbcb058959f6418b0da5365ccf21e06ba026762d8ed
Jul 29 00:02:53.894846 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:53.894820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"]
Jul 29 00:02:53.899753 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:02:53.899730    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b344c2b_d638_4e5c_8218_5fabe86f512f.slice/crio-8a5cf686a307fed8ae00e84abe534c16c9344afe93bff2c055f29a8bc0a750c6 WatchSource:0}: Error finding container 8a5cf686a307fed8ae00e84abe534c16c9344afe93bff2c055f29a8bc0a750c6: Status 404 returned error can't find the container with id 8a5cf686a307fed8ae00e84abe534c16c9344afe93bff2c055f29a8bc0a750c6
Jul 29 00:02:54.325342 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.325308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2" event={"ID":"6b344c2b-d638-4e5c-8218-5fabe86f512f","Type":"ContainerStarted","Data":"c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1"}
Jul 29 00:02:54.325552 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.325374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:02:54.325552 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.325400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2" event={"ID":"6b344c2b-d638-4e5c-8218-5fabe86f512f","Type":"ContainerStarted","Data":"8a5cf686a307fed8ae00e84abe534c16c9344afe93bff2c055f29a8bc0a750c6"}
Jul 29 00:02:54.326725 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.326700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725" event={"ID":"89c6c2d3-9727-4933-8f17-4fe98124ebaf","Type":"ContainerStarted","Data":"8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32"}
Jul 29 00:02:54.326853 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.326739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725" event={"ID":"89c6c2d3-9727-4933-8f17-4fe98124ebaf","Type":"ContainerStarted","Data":"b1875c97037b11c3ec9efbbcb058959f6418b0da5365ccf21e06ba026762d8ed"}
Jul 29 00:02:54.326853 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.326813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:02:54.342190 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.342147    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2" podStartSLOduration=1.342131764 podStartE2EDuration="1.342131764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:02:54.340750698 +0000 UTC m=+2469.843297240" watchObservedRunningTime="2026-07-29 00:02:54.342131764 +0000 UTC m=+2469.844678320"
Jul 29 00:02:54.356750 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:54.356701    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725" podStartSLOduration=1.356686793 podStartE2EDuration="1.356686793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:02:54.355807712 +0000 UTC m=+2469.858354242" watchObservedRunningTime="2026-07-29 00:02:54.356686793 +0000 UTC m=+2469.859233325"
Jul 29 00:02:59.216233 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:02:59.216190    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:03:05.333694 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:05.333664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:03:05.334135 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:05.333736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:03:22.210165 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.210133    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"]
Jul 29 00:03:22.213483 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.213461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.216010 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.215991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-48f20e-dockercfg-p2xw9\""
Jul 29 00:03:22.216139 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.216020    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-48f20e\""
Jul 29 00:03:22.224949 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.224921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"]
Jul 29 00:03:22.249397 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.249370    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"]
Jul 29 00:03:22.252794 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.252778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.262507 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.262486    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"]
Jul 29 00:03:22.370964 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.370935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9dx\" (UniqueName: \"kubernetes.io/projected/6b02bf3f-1e2f-486d-8861-4617631dada2-kube-api-access-ww9dx\") pod \"payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") " pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.371155 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.370984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0da6a-d18c-4a58-8055-ac988e12f312-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.371155 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.371057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b02bf3f-1e2f-486d-8861-4617631dada2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") " pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.371155 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.371134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/d3e0da6a-d18c-4a58-8055-ac988e12f312-kube-api-access-kwgpg\") pod \"payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.472544 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.472464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0da6a-d18c-4a58-8055-ac988e12f312-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.472544 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.472514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b02bf3f-1e2f-486d-8861-4617631dada2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") " pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.472786 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.472549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/d3e0da6a-d18c-4a58-8055-ac988e12f312-kube-api-access-kwgpg\") pod \"payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.472786 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.472597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9dx\" (UniqueName: \"kubernetes.io/projected/6b02bf3f-1e2f-486d-8861-4617631dada2-kube-api-access-ww9dx\") pod \"payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") " pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.473216 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.473194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b02bf3f-1e2f-486d-8861-4617631dada2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") " pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.473276 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.473220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0da6a-d18c-4a58-8055-ac988e12f312-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.480924 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.480902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9dx\" (UniqueName: \"kubernetes.io/projected/6b02bf3f-1e2f-486d-8861-4617631dada2-kube-api-access-ww9dx\") pod \"payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") " pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.481051 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.480978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/d3e0da6a-d18c-4a58-8055-ac988e12f312-kube-api-access-kwgpg\") pod \"payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.525921 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.525887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:22.563145 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.563105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:22.669410 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.669382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"]
Jul 29 00:03:22.671497 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:03:22.671468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e0da6a_d18c_4a58_8055_ac988e12f312.slice/crio-d77d03a637f811318f7c3bf501b7924c61d9201d97a43b9d41a59ce0e28b8019 WatchSource:0}: Error finding container d77d03a637f811318f7c3bf501b7924c61d9201d97a43b9d41a59ce0e28b8019: Status 404 returned error can't find the container with id d77d03a637f811318f7c3bf501b7924c61d9201d97a43b9d41a59ce0e28b8019
Jul 29 00:03:22.917303 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:22.917280    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"]
Jul 29 00:03:22.919938 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:03:22.919910    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b02bf3f_1e2f_486d_8861_4617631dada2.slice/crio-6c3e9cdcc82c73c0b4389ecc6d6901bc037a4dedf4eb5206376ba8337239c6d5 WatchSource:0}: Error finding container 6c3e9cdcc82c73c0b4389ecc6d6901bc037a4dedf4eb5206376ba8337239c6d5: Status 404 returned error can't find the container with id 6c3e9cdcc82c73c0b4389ecc6d6901bc037a4dedf4eb5206376ba8337239c6d5
Jul 29 00:03:23.465263 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.465228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6" event={"ID":"6b02bf3f-1e2f-486d-8861-4617631dada2","Type":"ContainerStarted","Data":"70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2"}
Jul 29 00:03:23.465754 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.465286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6" event={"ID":"6b02bf3f-1e2f-486d-8861-4617631dada2","Type":"ContainerStarted","Data":"6c3e9cdcc82c73c0b4389ecc6d6901bc037a4dedf4eb5206376ba8337239c6d5"}
Jul 29 00:03:23.465754 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.465326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:23.466753 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.466731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr" event={"ID":"d3e0da6a-d18c-4a58-8055-ac988e12f312","Type":"ContainerStarted","Data":"220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050"}
Jul 29 00:03:23.466850 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.466766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr" event={"ID":"d3e0da6a-d18c-4a58-8055-ac988e12f312","Type":"ContainerStarted","Data":"d77d03a637f811318f7c3bf501b7924c61d9201d97a43b9d41a59ce0e28b8019"}
Jul 29 00:03:23.466908 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.466854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:03:23.484242 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.484205    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6" podStartSLOduration=1.484192221 podStartE2EDuration="1.484192221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:03:23.482888672 +0000 UTC m=+2498.985435205" watchObservedRunningTime="2026-07-29 00:03:23.484192221 +0000 UTC m=+2498.986738751"
Jul 29 00:03:23.500479 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:23.500433    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr" podStartSLOduration=1.5004214299999998 podStartE2EDuration="1.50042143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:03:23.49849015 +0000 UTC m=+2499.001036681" watchObservedRunningTime="2026-07-29 00:03:23.50042143 +0000 UTC m=+2499.002967964"
Jul 29 00:03:28.716507 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:28.716469    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:03:34.480390 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:34.480360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:03:34.480864 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:03:34.480432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:04:04.735972 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:04.735740    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"]
Jul 29 00:04:04.736524 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:04.736095    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2" podUID="6b344c2b-d638-4e5c-8218-5fabe86f512f" containerName="payload-processing" containerID="cri-o://c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1" gracePeriod=30
Jul 29 00:04:04.743504 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:04.743478    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"]
Jul 29 00:04:04.743720 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:04.743699    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725" podUID="89c6c2d3-9727-4933-8f17-4fe98124ebaf" containerName="payload-pre-processing" containerID="cri-o://8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32" gracePeriod=30
Jul 29 00:04:05.000074 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.000010    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:04:05.031070 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.031043    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:04:05.152112 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.152070    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c6c2d3-9727-4933-8f17-4fe98124ebaf-plugins-config-volume\") pod \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") "
Jul 29 00:04:05.152112 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.152114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b344c2b-d638-4e5c-8218-5fabe86f512f-plugins-config-volume\") pod \"6b344c2b-d638-4e5c-8218-5fabe86f512f\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") "
Jul 29 00:04:05.152370 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.152147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwrjj\" (UniqueName: \"kubernetes.io/projected/6b344c2b-d638-4e5c-8218-5fabe86f512f-kube-api-access-gwrjj\") pod \"6b344c2b-d638-4e5c-8218-5fabe86f512f\" (UID: \"6b344c2b-d638-4e5c-8218-5fabe86f512f\") "
Jul 29 00:04:05.152370 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.152175    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmwmq\" (UniqueName: \"kubernetes.io/projected/89c6c2d3-9727-4933-8f17-4fe98124ebaf-kube-api-access-xmwmq\") pod \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\" (UID: \"89c6c2d3-9727-4933-8f17-4fe98124ebaf\") "
Jul 29 00:04:05.152527 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.152502    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b344c2b-d638-4e5c-8218-5fabe86f512f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b344c2b-d638-4e5c-8218-5fabe86f512f" (UID: "6b344c2b-d638-4e5c-8218-5fabe86f512f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:04:05.152596 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.152519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c6c2d3-9727-4933-8f17-4fe98124ebaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89c6c2d3-9727-4933-8f17-4fe98124ebaf" (UID: "89c6c2d3-9727-4933-8f17-4fe98124ebaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:04:05.154454 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.154424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c6c2d3-9727-4933-8f17-4fe98124ebaf-kube-api-access-xmwmq" (OuterVolumeSpecName: "kube-api-access-xmwmq") pod "89c6c2d3-9727-4933-8f17-4fe98124ebaf" (UID: "89c6c2d3-9727-4933-8f17-4fe98124ebaf"). InnerVolumeSpecName "kube-api-access-xmwmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:04:05.154454 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.154424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b344c2b-d638-4e5c-8218-5fabe86f512f-kube-api-access-gwrjj" (OuterVolumeSpecName: "kube-api-access-gwrjj") pod "6b344c2b-d638-4e5c-8218-5fabe86f512f" (UID: "6b344c2b-d638-4e5c-8218-5fabe86f512f"). InnerVolumeSpecName "kube-api-access-gwrjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:04:05.253146 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.253067    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c6c2d3-9727-4933-8f17-4fe98124ebaf-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:05.253146 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.253095    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b344c2b-d638-4e5c-8218-5fabe86f512f-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:05.253146 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.253106    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwrjj\" (UniqueName: \"kubernetes.io/projected/6b344c2b-d638-4e5c-8218-5fabe86f512f-kube-api-access-gwrjj\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:05.253146 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.253116    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwmq\" (UniqueName: \"kubernetes.io/projected/89c6c2d3-9727-4933-8f17-4fe98124ebaf-kube-api-access-xmwmq\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:05.677821 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.677788    2581 generic.go:356] "Generic (PLEG): container finished" podID="6b344c2b-d638-4e5c-8218-5fabe86f512f" containerID="c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1" exitCode=0
Jul 29 00:04:05.678040 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.677835    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"
Jul 29 00:04:05.678040 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.677880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2" event={"ID":"6b344c2b-d638-4e5c-8218-5fabe86f512f","Type":"ContainerDied","Data":"c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1"}
Jul 29 00:04:05.678040 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.677941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2" event={"ID":"6b344c2b-d638-4e5c-8218-5fabe86f512f","Type":"ContainerDied","Data":"8a5cf686a307fed8ae00e84abe534c16c9344afe93bff2c055f29a8bc0a750c6"}
Jul 29 00:04:05.678040 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.677972    2581 scope.go:117] "RemoveContainer" containerID="c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1"
Jul 29 00:04:05.679311 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.679291    2581 generic.go:356] "Generic (PLEG): container finished" podID="89c6c2d3-9727-4933-8f17-4fe98124ebaf" containerID="8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32" exitCode=0
Jul 29 00:04:05.679398 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.679341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"
Jul 29 00:04:05.679398 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.679369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725" event={"ID":"89c6c2d3-9727-4933-8f17-4fe98124ebaf","Type":"ContainerDied","Data":"8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32"}
Jul 29 00:04:05.679519 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.679443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725" event={"ID":"89c6c2d3-9727-4933-8f17-4fe98124ebaf","Type":"ContainerDied","Data":"b1875c97037b11c3ec9efbbcb058959f6418b0da5365ccf21e06ba026762d8ed"}
Jul 29 00:04:05.687212 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.687136    2581 scope.go:117] "RemoveContainer" containerID="c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1"
Jul 29 00:04:05.687468 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:04:05.687441    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1\": container with ID starting with c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1 not found: ID does not exist" containerID="c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1"
Jul 29 00:04:05.687585 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.687478    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1"} err="failed to get container status \"c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1\": rpc error: code = NotFound desc = could not find container \"c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1\": container with ID starting with c22284b448ef44d7e29feceb572138736e1abe5dc818f2f7d553fce7d576a2a1 not found: ID does not exist"
Jul 29 00:04:05.687585 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.687501    2581 scope.go:117] "RemoveContainer" containerID="8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32"
Jul 29 00:04:05.695525 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.695506    2581 scope.go:117] "RemoveContainer" containerID="8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32"
Jul 29 00:04:05.695819 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:04:05.695802    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32\": container with ID starting with 8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32 not found: ID does not exist" containerID="8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32"
Jul 29 00:04:05.695882 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.695829    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32"} err="failed to get container status \"8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32\": rpc error: code = NotFound desc = could not find container \"8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32\": container with ID starting with 8a04cb3470a367a63267013d18feabe74f23577dc35eec7d4e0fdec8ab5a2c32 not found: ID does not exist"
Jul 29 00:04:05.710012 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.703188    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"]
Jul 29 00:04:05.712396 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.712376    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-519e1e-668dcbf6b8-tw725"]
Jul 29 00:04:05.723595 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.723559    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"]
Jul 29 00:04:05.725175 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:05.725153    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-519e1e-54d5f444dd-qnjj2"]
Jul 29 00:04:06.981552 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:06.981515    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b344c2b-d638-4e5c-8218-5fabe86f512f" path="/var/lib/kubelet/pods/6b344c2b-d638-4e5c-8218-5fabe86f512f/volumes"
Jul 29 00:04:06.981971 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:06.981915    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c6c2d3-9727-4933-8f17-4fe98124ebaf" path="/var/lib/kubelet/pods/89c6c2d3-9727-4933-8f17-4fe98124ebaf/volumes"
Jul 29 00:04:07.324321 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:07.324234    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:04:27.143035 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.143002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"]
Jul 29 00:04:27.143494 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.143230    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6" podUID="6b02bf3f-1e2f-486d-8861-4617631dada2" containerName="payload-processing" containerID="cri-o://70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2" gracePeriod=30
Jul 29 00:04:27.148308 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.148280    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"]
Jul 29 00:04:27.148582 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.148509    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr" podUID="d3e0da6a-d18c-4a58-8055-ac988e12f312" containerName="payload-pre-processing" containerID="cri-o://220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050" gracePeriod=30
Jul 29 00:04:27.440177 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.440155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:04:27.443061 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.443041    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:04:27.448880 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.448862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww9dx\" (UniqueName: \"kubernetes.io/projected/6b02bf3f-1e2f-486d-8861-4617631dada2-kube-api-access-ww9dx\") pod \"6b02bf3f-1e2f-486d-8861-4617631dada2\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") "
Jul 29 00:04:27.448931 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.448902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0da6a-d18c-4a58-8055-ac988e12f312-plugins-config-volume\") pod \"d3e0da6a-d18c-4a58-8055-ac988e12f312\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") "
Jul 29 00:04:27.448980 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.448930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b02bf3f-1e2f-486d-8861-4617631dada2-plugins-config-volume\") pod \"6b02bf3f-1e2f-486d-8861-4617631dada2\" (UID: \"6b02bf3f-1e2f-486d-8861-4617631dada2\") "
Jul 29 00:04:27.449117 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.449097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/d3e0da6a-d18c-4a58-8055-ac988e12f312-kube-api-access-kwgpg\") pod \"d3e0da6a-d18c-4a58-8055-ac988e12f312\" (UID: \"d3e0da6a-d18c-4a58-8055-ac988e12f312\") "
Jul 29 00:04:27.449250 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.449225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b02bf3f-1e2f-486d-8861-4617631dada2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b02bf3f-1e2f-486d-8861-4617631dada2" (UID: "6b02bf3f-1e2f-486d-8861-4617631dada2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:04:27.449304 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.449240    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e0da6a-d18c-4a58-8055-ac988e12f312-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3e0da6a-d18c-4a58-8055-ac988e12f312" (UID: "d3e0da6a-d18c-4a58-8055-ac988e12f312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:04:27.449438 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.449424    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0da6a-d18c-4a58-8055-ac988e12f312-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:27.449496 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.449439    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b02bf3f-1e2f-486d-8861-4617631dada2-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:27.450949 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.450927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b02bf3f-1e2f-486d-8861-4617631dada2-kube-api-access-ww9dx" (OuterVolumeSpecName: "kube-api-access-ww9dx") pod "6b02bf3f-1e2f-486d-8861-4617631dada2" (UID: "6b02bf3f-1e2f-486d-8861-4617631dada2"). InnerVolumeSpecName "kube-api-access-ww9dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:04:27.451231 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.451215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e0da6a-d18c-4a58-8055-ac988e12f312-kube-api-access-kwgpg" (OuterVolumeSpecName: "kube-api-access-kwgpg") pod "d3e0da6a-d18c-4a58-8055-ac988e12f312" (UID: "d3e0da6a-d18c-4a58-8055-ac988e12f312"). InnerVolumeSpecName "kube-api-access-kwgpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:04:27.550712 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.550679    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/d3e0da6a-d18c-4a58-8055-ac988e12f312-kube-api-access-kwgpg\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:27.550712 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.550705    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww9dx\" (UniqueName: \"kubernetes.io/projected/6b02bf3f-1e2f-486d-8861-4617631dada2-kube-api-access-ww9dx\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:04:27.785724 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.785610    2581 generic.go:356] "Generic (PLEG): container finished" podID="d3e0da6a-d18c-4a58-8055-ac988e12f312" containerID="220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050" exitCode=0
Jul 29 00:04:27.785724 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.785679    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"
Jul 29 00:04:27.785724 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.785697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr" event={"ID":"d3e0da6a-d18c-4a58-8055-ac988e12f312","Type":"ContainerDied","Data":"220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050"}
Jul 29 00:04:27.786000 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.785753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr" event={"ID":"d3e0da6a-d18c-4a58-8055-ac988e12f312","Type":"ContainerDied","Data":"d77d03a637f811318f7c3bf501b7924c61d9201d97a43b9d41a59ce0e28b8019"}
Jul 29 00:04:27.786000 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.785781    2581 scope.go:117] "RemoveContainer" containerID="220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050"
Jul 29 00:04:27.787077 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.787053    2581 generic.go:356] "Generic (PLEG): container finished" podID="6b02bf3f-1e2f-486d-8861-4617631dada2" containerID="70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2" exitCode=0
Jul 29 00:04:27.787160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.787100    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"
Jul 29 00:04:27.787211 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.787148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6" event={"ID":"6b02bf3f-1e2f-486d-8861-4617631dada2","Type":"ContainerDied","Data":"70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2"}
Jul 29 00:04:27.787211 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.787202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6" event={"ID":"6b02bf3f-1e2f-486d-8861-4617631dada2","Type":"ContainerDied","Data":"6c3e9cdcc82c73c0b4389ecc6d6901bc037a4dedf4eb5206376ba8337239c6d5"}
Jul 29 00:04:27.794341 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.794324    2581 scope.go:117] "RemoveContainer" containerID="220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050"
Jul 29 00:04:27.794652 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:04:27.794611    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050\": container with ID starting with 220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050 not found: ID does not exist" containerID="220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050"
Jul 29 00:04:27.794730 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.794661    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050"} err="failed to get container status \"220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050\": rpc error: code = NotFound desc = could not find container \"220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050\": container with ID starting with 220b57a0fe9db47b6429dce16e71090cc7d4cb854f00a1370e96f310cf104050 not found: ID does not exist"
Jul 29 00:04:27.794730 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.794678    2581 scope.go:117] "RemoveContainer" containerID="70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2"
Jul 29 00:04:27.802722 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.802706    2581 scope.go:117] "RemoveContainer" containerID="70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2"
Jul 29 00:04:27.802958 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:04:27.802940    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2\": container with ID starting with 70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2 not found: ID does not exist" containerID="70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2"
Jul 29 00:04:27.803007 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.802964    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2"} err="failed to get container status \"70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2\": rpc error: code = NotFound desc = could not find container \"70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2\": container with ID starting with 70c3b64e43975e3e367503af15e04bf4f37f79bbfd9d9cf069be8d99ce58c0e2 not found: ID does not exist"
Jul 29 00:04:27.810766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.810743    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"]
Jul 29 00:04:27.814477 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.814456    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48f20e-786ff9b87b-qntkr"]
Jul 29 00:04:27.823793 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.823771    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"]
Jul 29 00:04:27.827177 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:27.827158    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48f20e-57db5fc744-j8vp6"]
Jul 29 00:04:28.981488 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:28.981455    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b02bf3f-1e2f-486d-8861-4617631dada2" path="/var/lib/kubelet/pods/6b02bf3f-1e2f-486d-8861-4617631dada2/volumes"
Jul 29 00:04:28.981880 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:28.981830    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e0da6a-d18c-4a58-8055-ac988e12f312" path="/var/lib/kubelet/pods/d3e0da6a-d18c-4a58-8055-ac988e12f312/volumes"
Jul 29 00:04:32.111458 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:32.111426    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:04:37.953972 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.953932    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"]
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b344c2b-d638-4e5c-8218-5fabe86f512f" containerName="payload-processing"
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954463    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b344c2b-d638-4e5c-8218-5fabe86f512f" containerName="payload-processing"
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954486    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b02bf3f-1e2f-486d-8861-4617631dada2" containerName="payload-processing"
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954495    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b02bf3f-1e2f-486d-8861-4617631dada2" containerName="payload-processing"
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954506    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c6c2d3-9727-4933-8f17-4fe98124ebaf" containerName="payload-pre-processing"
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954516    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c6c2d3-9727-4933-8f17-4fe98124ebaf" containerName="payload-pre-processing"
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954531    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e0da6a-d18c-4a58-8055-ac988e12f312" containerName="payload-pre-processing"
Jul 29 00:04:37.954586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954541    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e0da6a-d18c-4a58-8055-ac988e12f312" containerName="payload-pre-processing"
Jul 29 00:04:37.955035 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954657    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b344c2b-d638-4e5c-8218-5fabe86f512f" containerName="payload-processing"
Jul 29 00:04:37.955035 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954672    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e0da6a-d18c-4a58-8055-ac988e12f312" containerName="payload-pre-processing"
Jul 29 00:04:37.955035 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954685    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c6c2d3-9727-4933-8f17-4fe98124ebaf" containerName="payload-pre-processing"
Jul 29 00:04:37.955035 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.954695    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b02bf3f-1e2f-486d-8861-4617631dada2" containerName="payload-processing"
Jul 29 00:04:37.959067 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.959046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:37.961535 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.961516    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-daedc5\""
Jul 29 00:04:37.967863 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.967839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"]
Jul 29 00:04:37.989904 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.989878    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"]
Jul 29 00:04:37.993394 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.993373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:37.996390 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:37.996372    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-daedc5-dockercfg-rdwvd\""
Jul 29 00:04:38.003589 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.003567    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"]
Jul 29 00:04:38.043569 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.043524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdj5t\" (UniqueName: \"kubernetes.io/projected/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-kube-api-access-jdj5t\") pod \"payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.043803 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.043582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.043803 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.043691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmcwx\" (UniqueName: \"kubernetes.io/projected/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-kube-api-access-vmcwx\") pod \"payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.043803 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.043726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.144888 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.144839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdj5t\" (UniqueName: \"kubernetes.io/projected/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-kube-api-access-jdj5t\") pod \"payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.145105 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.144905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.145105 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.144960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmcwx\" (UniqueName: \"kubernetes.io/projected/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-kube-api-access-vmcwx\") pod \"payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.145105 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.144992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.145707 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.145684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.145789 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.145735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.153326 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.153292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmcwx\" (UniqueName: \"kubernetes.io/projected/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-kube-api-access-vmcwx\") pod \"payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.153326 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.153298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdj5t\" (UniqueName: \"kubernetes.io/projected/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-kube-api-access-jdj5t\") pod \"payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.270706 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.270585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.303344 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.303307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.621618 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.621534    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"]
Jul 29 00:04:38.623976 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:04:38.623945    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f34dc0_5b9c_4fc5_9f15_83b74a0984a1.slice/crio-04e83911ec4fe09c14b30cd798cd479ccb4aad071fa7c9d5843a8af824b2e740 WatchSource:0}: Error finding container 04e83911ec4fe09c14b30cd798cd479ccb4aad071fa7c9d5843a8af824b2e740: Status 404 returned error can't find the container with id 04e83911ec4fe09c14b30cd798cd479ccb4aad071fa7c9d5843a8af824b2e740
Jul 29 00:04:38.655123 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.655093    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"]
Jul 29 00:04:38.656955 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:04:38.656917    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0ddcf9_eb72_4d24_8ce8_7acb01019a28.slice/crio-b14125867f34b25f5d43d62cc6218412a1ea96f3a0289790c88c03a50a41766a WatchSource:0}: Error finding container b14125867f34b25f5d43d62cc6218412a1ea96f3a0289790c88c03a50a41766a: Status 404 returned error can't find the container with id b14125867f34b25f5d43d62cc6218412a1ea96f3a0289790c88c03a50a41766a
Jul 29 00:04:38.840048 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.840011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4" event={"ID":"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1","Type":"ContainerStarted","Data":"31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79"}
Jul 29 00:04:38.840256 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.840069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4" event={"ID":"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1","Type":"ContainerStarted","Data":"04e83911ec4fe09c14b30cd798cd479ccb4aad071fa7c9d5843a8af824b2e740"}
Jul 29 00:04:38.840256 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.840096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:04:38.841594 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.841569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l" event={"ID":"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28","Type":"ContainerStarted","Data":"1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33"}
Jul 29 00:04:38.841705 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.841613    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:38.841705 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.841653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l" event={"ID":"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28","Type":"ContainerStarted","Data":"b14125867f34b25f5d43d62cc6218412a1ea96f3a0289790c88c03a50a41766a"}
Jul 29 00:04:38.855728 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.855685    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4" podStartSLOduration=1.855668645 podStartE2EDuration="1.855668645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:04:38.854160768 +0000 UTC m=+2574.356707300" watchObservedRunningTime="2026-07-29 00:04:38.855668645 +0000 UTC m=+2574.358215175"
Jul 29 00:04:38.870222 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:38.870171    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l" podStartSLOduration=1.870154576 podStartE2EDuration="1.870154576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:04:38.868724384 +0000 UTC m=+2574.371270915" watchObservedRunningTime="2026-07-29 00:04:38.870154576 +0000 UTC m=+2574.372701120"
Jul 29 00:04:43.319642 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:43.319597    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:04:49.847875 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:49.847844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:04:49.848354 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:04:49.848055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:05:06.586240 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.586206    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"]
Jul 29 00:05:06.589787 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.589769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.592288 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.592269    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-fa6130\""
Jul 29 00:05:06.592413 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.592290    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-fa6130-dockercfg-tfs5z\""
Jul 29 00:05:06.600815 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.600788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"]
Jul 29 00:05:06.639461 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.639434    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"]
Jul 29 00:05:06.642883 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.642863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:06.653843 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.653815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"]
Jul 29 00:05:06.707589 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.707559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgprr\" (UniqueName: \"kubernetes.io/projected/5a4cabbe-c867-48fe-93da-84e081f5eb13-kube-api-access-cgprr\") pod \"payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.707737 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.707640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4cabbe-c867-48fe-93da-84e081f5eb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.808694 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.808658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgprr\" (UniqueName: \"kubernetes.io/projected/5a4cabbe-c867-48fe-93da-84e081f5eb13-kube-api-access-cgprr\") pod \"payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.808694 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.808697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flwrc\" (UniqueName: \"kubernetes.io/projected/5f6a28c7-0130-4df3-99ef-0679a09f9399-kube-api-access-flwrc\") pod \"payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") " pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:06.808907 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.808770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4cabbe-c867-48fe-93da-84e081f5eb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.808907 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.808849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6a28c7-0130-4df3-99ef-0679a09f9399-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") " pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:06.809479 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.809460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4cabbe-c867-48fe-93da-84e081f5eb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.816814 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.816788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgprr\" (UniqueName: \"kubernetes.io/projected/5a4cabbe-c867-48fe-93da-84e081f5eb13-kube-api-access-cgprr\") pod \"payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.900643 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.900592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:06.910062 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.910036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flwrc\" (UniqueName: \"kubernetes.io/projected/5f6a28c7-0130-4df3-99ef-0679a09f9399-kube-api-access-flwrc\") pod \"payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") " pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:06.910174 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.910134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6a28c7-0130-4df3-99ef-0679a09f9399-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") " pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:06.910996 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.910967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6a28c7-0130-4df3-99ef-0679a09f9399-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") " pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:06.918551 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.918528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flwrc\" (UniqueName: \"kubernetes.io/projected/5f6a28c7-0130-4df3-99ef-0679a09f9399-kube-api-access-flwrc\") pod \"payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") " pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:06.952939 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:06.952750    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:07.045506 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.045449    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"]
Jul 29 00:05:07.049004 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:05:07.048950    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4cabbe_c867_48fe_93da_84e081f5eb13.slice/crio-d1a17c5eb5280378edb68319611d58d3bc5964713fe66e99dce4625dba07e0be WatchSource:0}: Error finding container d1a17c5eb5280378edb68319611d58d3bc5964713fe66e99dce4625dba07e0be: Status 404 returned error can't find the container with id d1a17c5eb5280378edb68319611d58d3bc5964713fe66e99dce4625dba07e0be
Jul 29 00:05:07.097815 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.097790    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"]
Jul 29 00:05:07.100781 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:05:07.100754    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6a28c7_0130_4df3_99ef_0679a09f9399.slice/crio-938706a3e7ef2b915d90d1c4041b8137c365cca61373494ec878ae5a4647b4db WatchSource:0}: Error finding container 938706a3e7ef2b915d90d1c4041b8137c365cca61373494ec878ae5a4647b4db: Status 404 returned error can't find the container with id 938706a3e7ef2b915d90d1c4041b8137c365cca61373494ec878ae5a4647b4db
Jul 29 00:05:07.989133 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.989079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs" event={"ID":"5f6a28c7-0130-4df3-99ef-0679a09f9399","Type":"ContainerStarted","Data":"adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343"}
Jul 29 00:05:07.989622 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.989226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs" event={"ID":"5f6a28c7-0130-4df3-99ef-0679a09f9399","Type":"ContainerStarted","Data":"938706a3e7ef2b915d90d1c4041b8137c365cca61373494ec878ae5a4647b4db"}
Jul 29 00:05:07.989622 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.989265    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:07.990521 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.990496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7" event={"ID":"5a4cabbe-c867-48fe-93da-84e081f5eb13","Type":"ContainerStarted","Data":"e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b"}
Jul 29 00:05:07.990600 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.990537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7" event={"ID":"5a4cabbe-c867-48fe-93da-84e081f5eb13","Type":"ContainerStarted","Data":"d1a17c5eb5280378edb68319611d58d3bc5964713fe66e99dce4625dba07e0be"}
Jul 29 00:05:07.990600 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:07.990569    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:08.004895 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:08.004850    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs" podStartSLOduration=2.004836632 podStartE2EDuration="2.004836632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:05:08.003332368 +0000 UTC m=+2603.505878900" watchObservedRunningTime="2026-07-29 00:05:08.004836632 +0000 UTC m=+2603.507383162"
Jul 29 00:05:08.018344 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:08.018304    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7" podStartSLOduration=2.018290644 podStartE2EDuration="2.018290644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:05:08.016642945 +0000 UTC m=+2603.519189468" watchObservedRunningTime="2026-07-29 00:05:08.018290644 +0000 UTC m=+2603.520837243"
Jul 29 00:05:13.711829 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:13.711792    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:05:18.997337 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:18.997306    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:05:18.997765 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:18.997421    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:05:47.828168 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.828131    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"]
Jul 29 00:05:47.831940 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.831923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:47.835763 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.835735    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"openshift-service-ca.crt\""
Jul 29 00:05:47.835763 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.835744    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"default-dockercfg-wt7lm\""
Jul 29 00:05:47.835986 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.835835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"kube-root-ca.crt\""
Jul 29 00:05:47.835986 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.835859    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"auth-test-model-daedc5-kserve-self-signed-certs\""
Jul 29 00:05:47.841097 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.841075    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"]
Jul 29 00:05:47.986397 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.986359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-dshm\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:47.986397 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.986398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-model-cache\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:47.986602 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.986419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-home\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:47.986602 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.986437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-tmp-dir\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:47.986602 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.986460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0175fb-d112-44fe-a046-fc5bcb602a54-tls-certs\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:47.986602 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:47.986488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s447g\" (UniqueName: \"kubernetes.io/projected/7d0175fb-d112-44fe-a046-fc5bcb602a54-kube-api-access-s447g\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088065 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.087977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s447g\" (UniqueName: \"kubernetes.io/projected/7d0175fb-d112-44fe-a046-fc5bcb602a54-kube-api-access-s447g\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-dshm\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-model-cache\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-home\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-tmp-dir\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0175fb-d112-44fe-a046-fc5bcb602a54-tls-certs\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088603 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-model-cache\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088708 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-home\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.088708 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.088692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-tmp-dir\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.090381 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.090361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-dshm\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.090684 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.090666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0175fb-d112-44fe-a046-fc5bcb602a54-tls-certs\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.100021 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.100000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s447g\" (UniqueName: \"kubernetes.io/projected/7d0175fb-d112-44fe-a046-fc5bcb602a54-kube-api-access-s447g\") pod \"auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") " pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.142062 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.142029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:48.278193 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:48.278165    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"]
Jul 29 00:05:48.280180 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:05:48.280146    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d0175fb_d112_44fe_a046_fc5bcb602a54.slice/crio-93fefb2e89e236ed734ce6154fa17ee7169b7c14733c03c28e86c8c619a06308 WatchSource:0}: Error finding container 93fefb2e89e236ed734ce6154fa17ee7169b7c14733c03c28e86c8c619a06308: Status 404 returned error can't find the container with id 93fefb2e89e236ed734ce6154fa17ee7169b7c14733c03c28e86c8c619a06308
Jul 29 00:05:49.189731 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:49.189691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh" event={"ID":"7d0175fb-d112-44fe-a046-fc5bcb602a54","Type":"ContainerStarted","Data":"72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd"}
Jul 29 00:05:49.189731 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:49.189739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh" event={"ID":"7d0175fb-d112-44fe-a046-fc5bcb602a54","Type":"ContainerStarted","Data":"93fefb2e89e236ed734ce6154fa17ee7169b7c14733c03c28e86c8c619a06308"}
Jul 29 00:05:49.205969 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:49.205924    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh" podStartSLOduration=2.205908347 podStartE2EDuration="2.205908347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:05:49.204239267 +0000 UTC m=+2644.706785797" watchObservedRunningTime="2026-07-29 00:05:49.205908347 +0000 UTC m=+2644.708454895"
Jul 29 00:05:51.511981 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:51.511941    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:05:58.142718 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:58.142681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:58.142718 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:58.142728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:58.155483 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:58.155458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:05:58.242966 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:05:58.242938    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:06:36.719284 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.719251    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"]
Jul 29 00:06:36.722596 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.722581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.725254 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.725234    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"kube-root-ca.crt\""
Jul 29 00:06:36.726246 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.726231    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"openshift-service-ca.crt\""
Jul 29 00:06:36.726314 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.726244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"default-dockercfg-7xbrr\""
Jul 29 00:06:36.726392 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.726346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"auth-test-model-fa6130-kserve-self-signed-certs\""
Jul 29 00:06:36.732852 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.732830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"]
Jul 29 00:06:36.819643 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.819591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-home\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.819801 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.819651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e56807-8cae-4168-8348-4aac18b82ecd-tls-certs\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.819801 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.819684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-dshm\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.819801 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.819698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-model-cache\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.819801 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.819779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6g8\" (UniqueName: \"kubernetes.io/projected/b3e56807-8cae-4168-8348-4aac18b82ecd-kube-api-access-pc6g8\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.819985 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.819803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-tmp-dir\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921163 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6g8\" (UniqueName: \"kubernetes.io/projected/b3e56807-8cae-4168-8348-4aac18b82ecd-kube-api-access-pc6g8\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921163 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-tmp-dir\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921431 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-home\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921431 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e56807-8cae-4168-8348-4aac18b82ecd-tls-certs\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921431 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-dshm\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921431 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-model-cache\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921716 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-tmp-dir\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921777 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-home\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.921777 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.921767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-model-cache\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.923527 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.923507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-dshm\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.923853 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.923833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e56807-8cae-4168-8348-4aac18b82ecd-tls-certs\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:36.929427 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:36.929406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6g8\" (UniqueName: \"kubernetes.io/projected/b3e56807-8cae-4168-8348-4aac18b82ecd-kube-api-access-pc6g8\") pod \"auth-test-model-fa6130-kserve-7b95d778f5-6tkwj\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") " pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:37.032335 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:37.032242    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:37.164792 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:37.164764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"]
Jul 29 00:06:37.167374 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:06:37.167342    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3e56807_8cae_4168_8348_4aac18b82ecd.slice/crio-9b8fc57cc9dd6cdee0d033c470fa26dbe670db7c8e621d3583be0183f62e70e0 WatchSource:0}: Error finding container 9b8fc57cc9dd6cdee0d033c470fa26dbe670db7c8e621d3583be0183f62e70e0: Status 404 returned error can't find the container with id 9b8fc57cc9dd6cdee0d033c470fa26dbe670db7c8e621d3583be0183f62e70e0
Jul 29 00:06:37.425499 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:37.425412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj" event={"ID":"b3e56807-8cae-4168-8348-4aac18b82ecd","Type":"ContainerStarted","Data":"4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab"}
Jul 29 00:06:37.425499 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:37.425463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj" event={"ID":"b3e56807-8cae-4168-8348-4aac18b82ecd","Type":"ContainerStarted","Data":"9b8fc57cc9dd6cdee0d033c470fa26dbe670db7c8e621d3583be0183f62e70e0"}
Jul 29 00:06:37.441766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:37.441721    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj" podStartSLOduration=1.44170419 podStartE2EDuration="1.44170419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:06:37.441136016 +0000 UTC m=+2692.943682546" watchObservedRunningTime="2026-07-29 00:06:37.44170419 +0000 UTC m=+2692.944250721"
Jul 29 00:06:40.416152 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:40.416121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:06:47.032685 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:47.032646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:47.033105 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:47.032714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:47.045068 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:47.045045    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:06:47.487508 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:06:47.487476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:07:08.209611 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:07:08.209579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:07:28.810262 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:07:28.810218    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:07:47.521075 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:07:47.521039    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:08:08.614000 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:08:08.613918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:08:22.512389 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:08:22.512348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:08:42.845797 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:08:42.845756    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:09:01.019349 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:09:01.019311    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:09:22.714004 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:09:22.713968    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:09:39.617480 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:09:39.617379    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:09:53.029072 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:09:53.029036    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:10:14.223809 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:10:14.223775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:10:28.518672 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:10:28.518619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:10:40.119944 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:10:40.119898    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:10:56.014057 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:10:56.014022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:11:00.381008 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:00.380975    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"]
Jul 29 00:11:01.332017 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:01.331984    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"]
Jul 29 00:11:11.525590 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:11.525549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:11:15.399767 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.399718    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh" podUID="7d0175fb-d112-44fe-a046-fc5bcb602a54" containerName="main" containerID="cri-o://72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd" gracePeriod=45
Jul 29 00:11:15.647280 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.647254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:11:15.698453 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698371    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s447g\" (UniqueName: \"kubernetes.io/projected/7d0175fb-d112-44fe-a046-fc5bcb602a54-kube-api-access-s447g\") pod \"7d0175fb-d112-44fe-a046-fc5bcb602a54\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") "
Jul 29 00:11:15.698453 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-tmp-dir\") pod \"7d0175fb-d112-44fe-a046-fc5bcb602a54\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") "
Jul 29 00:11:15.698714 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-model-cache\") pod \"7d0175fb-d112-44fe-a046-fc5bcb602a54\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") "
Jul 29 00:11:15.698714 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0175fb-d112-44fe-a046-fc5bcb602a54-tls-certs\") pod \"7d0175fb-d112-44fe-a046-fc5bcb602a54\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") "
Jul 29 00:11:15.698714 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-home\") pod \"7d0175fb-d112-44fe-a046-fc5bcb602a54\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") "
Jul 29 00:11:15.698714 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-dshm\") pod \"7d0175fb-d112-44fe-a046-fc5bcb602a54\" (UID: \"7d0175fb-d112-44fe-a046-fc5bcb602a54\") "
Jul 29 00:11:15.698903 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698755    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d0175fb-d112-44fe-a046-fc5bcb602a54" (UID: "7d0175fb-d112-44fe-a046-fc5bcb602a54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:15.698903 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698758    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d0175fb-d112-44fe-a046-fc5bcb602a54" (UID: "7d0175fb-d112-44fe-a046-fc5bcb602a54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:15.698903 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698808    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-home" (OuterVolumeSpecName: "home") pod "7d0175fb-d112-44fe-a046-fc5bcb602a54" (UID: "7d0175fb-d112-44fe-a046-fc5bcb602a54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:15.699054 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698912    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:15.699054 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698933    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:15.699054 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.698946    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:15.700803 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.700767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-dshm" (OuterVolumeSpecName: "dshm") pod "7d0175fb-d112-44fe-a046-fc5bcb602a54" (UID: "7d0175fb-d112-44fe-a046-fc5bcb602a54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:15.700910 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.700828    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0175fb-d112-44fe-a046-fc5bcb602a54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d0175fb-d112-44fe-a046-fc5bcb602a54" (UID: "7d0175fb-d112-44fe-a046-fc5bcb602a54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:11:15.700910 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.700897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0175fb-d112-44fe-a046-fc5bcb602a54-kube-api-access-s447g" (OuterVolumeSpecName: "kube-api-access-s447g") pod "7d0175fb-d112-44fe-a046-fc5bcb602a54" (UID: "7d0175fb-d112-44fe-a046-fc5bcb602a54"). InnerVolumeSpecName "kube-api-access-s447g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:11:15.775216 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.775185    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d0175fb-d112-44fe-a046-fc5bcb602a54" containerID="72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd" exitCode=0
Jul 29 00:11:15.775394 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.775226    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"
Jul 29 00:11:15.775394 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.775272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh" event={"ID":"7d0175fb-d112-44fe-a046-fc5bcb602a54","Type":"ContainerDied","Data":"72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd"}
Jul 29 00:11:15.775394 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.775314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh" event={"ID":"7d0175fb-d112-44fe-a046-fc5bcb602a54","Type":"ContainerDied","Data":"93fefb2e89e236ed734ce6154fa17ee7169b7c14733c03c28e86c8c619a06308"}
Jul 29 00:11:15.775394 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.775333    2581 scope.go:117] "RemoveContainer" containerID="72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd"
Jul 29 00:11:15.784571 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.784555    2581 scope.go:117] "RemoveContainer" containerID="72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd"
Jul 29 00:11:15.784917 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:11:15.784898    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd\": container with ID starting with 72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd not found: ID does not exist" containerID="72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd"
Jul 29 00:11:15.784972 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.784925    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd"} err="failed to get container status \"72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd\": rpc error: code = NotFound desc = could not find container \"72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd\": container with ID starting with 72465a58e1c68258eb87bc7c938cde358f0d6e07450bcf08aa1c952055dd0bdd not found: ID does not exist"
Jul 29 00:11:15.797779 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.797751    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"]
Jul 29 00:11:15.800165 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.800146    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d0175fb-d112-44fe-a046-fc5bcb602a54-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:15.800265 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.800170    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s447g\" (UniqueName: \"kubernetes.io/projected/7d0175fb-d112-44fe-a046-fc5bcb602a54-kube-api-access-s447g\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:15.800265 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.800187    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d0175fb-d112-44fe-a046-fc5bcb602a54-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:15.803057 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:15.803038    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/auth-test-model-daedc5-kserve-6f6bc79df9-4xqsh"]
Jul 29 00:11:16.352093 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.352039    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj" podUID="b3e56807-8cae-4168-8348-4aac18b82ecd" containerName="main" containerID="cri-o://4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab" gracePeriod=45
Jul 29 00:11:16.598436 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.598412    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:11:16.708790 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.708759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-home\") pod \"b3e56807-8cae-4168-8348-4aac18b82ecd\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") "
Jul 29 00:11:16.708981 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.708815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-dshm\") pod \"b3e56807-8cae-4168-8348-4aac18b82ecd\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") "
Jul 29 00:11:16.708981 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.708832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-model-cache\") pod \"b3e56807-8cae-4168-8348-4aac18b82ecd\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") "
Jul 29 00:11:16.708981 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.708867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc6g8\" (UniqueName: \"kubernetes.io/projected/b3e56807-8cae-4168-8348-4aac18b82ecd-kube-api-access-pc6g8\") pod \"b3e56807-8cae-4168-8348-4aac18b82ecd\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") "
Jul 29 00:11:16.708981 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.708899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e56807-8cae-4168-8348-4aac18b82ecd-tls-certs\") pod \"b3e56807-8cae-4168-8348-4aac18b82ecd\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") "
Jul 29 00:11:16.708981 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.708952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-tmp-dir\") pod \"b3e56807-8cae-4168-8348-4aac18b82ecd\" (UID: \"b3e56807-8cae-4168-8348-4aac18b82ecd\") "
Jul 29 00:11:16.709259 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.709093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-home" (OuterVolumeSpecName: "home") pod "b3e56807-8cae-4168-8348-4aac18b82ecd" (UID: "b3e56807-8cae-4168-8348-4aac18b82ecd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:16.709259 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.709124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3e56807-8cae-4168-8348-4aac18b82ecd" (UID: "b3e56807-8cae-4168-8348-4aac18b82ecd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:16.709259 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.709257    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:16.709421 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.709270    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:16.709421 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.709291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3e56807-8cae-4168-8348-4aac18b82ecd" (UID: "b3e56807-8cae-4168-8348-4aac18b82ecd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:16.711139 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.711108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-dshm" (OuterVolumeSpecName: "dshm") pod "b3e56807-8cae-4168-8348-4aac18b82ecd" (UID: "b3e56807-8cae-4168-8348-4aac18b82ecd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:11:16.711236 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.711159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e56807-8cae-4168-8348-4aac18b82ecd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3e56807-8cae-4168-8348-4aac18b82ecd" (UID: "b3e56807-8cae-4168-8348-4aac18b82ecd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:11:16.711236 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.711166    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e56807-8cae-4168-8348-4aac18b82ecd-kube-api-access-pc6g8" (OuterVolumeSpecName: "kube-api-access-pc6g8") pod "b3e56807-8cae-4168-8348-4aac18b82ecd" (UID: "b3e56807-8cae-4168-8348-4aac18b82ecd"). InnerVolumeSpecName "kube-api-access-pc6g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:11:16.781728 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.781695    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3e56807-8cae-4168-8348-4aac18b82ecd" containerID="4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab" exitCode=0
Jul 29 00:11:16.781872 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.781737    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"
Jul 29 00:11:16.781872 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.781772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj" event={"ID":"b3e56807-8cae-4168-8348-4aac18b82ecd","Type":"ContainerDied","Data":"4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab"}
Jul 29 00:11:16.781872 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.781814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj" event={"ID":"b3e56807-8cae-4168-8348-4aac18b82ecd","Type":"ContainerDied","Data":"9b8fc57cc9dd6cdee0d033c470fa26dbe670db7c8e621d3583be0183f62e70e0"}
Jul 29 00:11:16.781872 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.781832    2581 scope.go:117] "RemoveContainer" containerID="4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab"
Jul 29 00:11:16.790495 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.790476    2581 scope.go:117] "RemoveContainer" containerID="4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab"
Jul 29 00:11:16.790761 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:11:16.790742    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab\": container with ID starting with 4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab not found: ID does not exist" containerID="4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab"
Jul 29 00:11:16.790830 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.790768    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab"} err="failed to get container status \"4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab\": rpc error: code = NotFound desc = could not find container \"4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab\": container with ID starting with 4e34359e8286b4abee6e51079f076d019023b07dc9f358e7053d675d8d6665ab not found: ID does not exist"
Jul 29 00:11:16.805244 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.805221    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"]
Jul 29 00:11:16.809746 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.809727    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e56807-8cae-4168-8348-4aac18b82ecd-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:16.809839 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.809749    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:16.809839 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.809761    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e56807-8cae-4168-8348-4aac18b82ecd-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:16.809839 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.809774    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6g8\" (UniqueName: \"kubernetes.io/projected/b3e56807-8cae-4168-8348-4aac18b82ecd-kube-api-access-pc6g8\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:11:16.810330 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.810311    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/auth-test-model-fa6130-kserve-7b95d778f5-6tkwj"]
Jul 29 00:11:16.982712 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.982618    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0175fb-d112-44fe-a046-fc5bcb602a54" path="/var/lib/kubelet/pods/7d0175fb-d112-44fe-a046-fc5bcb602a54/volumes"
Jul 29 00:11:16.983049 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:16.983035    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3e56807-8cae-4168-8348-4aac18b82ecd" path="/var/lib/kubelet/pods/b3e56807-8cae-4168-8348-4aac18b82ecd/volumes"
Jul 29 00:11:17.703655 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.703612    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"]
Jul 29 00:11:17.704029 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.704011    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e56807-8cae-4168-8348-4aac18b82ecd" containerName="main"
Jul 29 00:11:17.704029 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.704025    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e56807-8cae-4168-8348-4aac18b82ecd" containerName="main"
Jul 29 00:11:17.704110 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.704045    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0175fb-d112-44fe-a046-fc5bcb602a54" containerName="main"
Jul 29 00:11:17.704110 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.704053    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0175fb-d112-44fe-a046-fc5bcb602a54" containerName="main"
Jul 29 00:11:17.704173 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.704126    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0175fb-d112-44fe-a046-fc5bcb602a54" containerName="main"
Jul 29 00:11:17.704173 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.704141    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e56807-8cae-4168-8348-4aac18b82ecd" containerName="main"
Jul 29 00:11:17.708494 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.708476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.718046 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.717907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"openshift-service-ca.crt\""
Jul 29 00:11:17.718046 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.717936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"default-dockercfg-wt7lm\""
Jul 29 00:11:17.718046 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.717945    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"sub-test-model-daedc5-kserve-self-signed-certs\""
Jul 29 00:11:17.718046 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.717954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"kube-root-ca.crt\""
Jul 29 00:11:17.730703 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.730684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"]
Jul 29 00:11:17.818051 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.818016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-tmp-dir\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.818051 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.818047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85541bc-24b6-4645-9f43-7b150b724c1e-tls-certs\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.818317 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.818176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-model-cache\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.818317 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.818239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-home\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.818317 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.818293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-dshm\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.818454 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.818331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqk9\" (UniqueName: \"kubernetes.io/projected/b85541bc-24b6-4645-9f43-7b150b724c1e-kube-api-access-sxqk9\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919121 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-home\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919317 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-dshm\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919317 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqk9\" (UniqueName: \"kubernetes.io/projected/b85541bc-24b6-4645-9f43-7b150b724c1e-kube-api-access-sxqk9\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919317 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-tmp-dir\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919493 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85541bc-24b6-4645-9f43-7b150b724c1e-tls-certs\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919493 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-model-cache\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919594 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-home\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919673 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-tmp-dir\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.919822 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.919793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-model-cache\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.921393 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.921376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-dshm\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.921811 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.921797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85541bc-24b6-4645-9f43-7b150b724c1e-tls-certs\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:17.931050 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:17.931027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqk9\" (UniqueName: \"kubernetes.io/projected/b85541bc-24b6-4645-9f43-7b150b724c1e-kube-api-access-sxqk9\") pod \"sub-test-model-daedc5-kserve-5945b486dc-4cns4\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") " pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:18.018463 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:18.018375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:18.158307 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:18.158109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"]
Jul 29 00:11:18.160547 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:11:18.160516    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb85541bc_24b6_4645_9f43_7b150b724c1e.slice/crio-e9e1d739e45130f3bf35b6428c9485a01762eccdd4c50d3fb139e46addc52860 WatchSource:0}: Error finding container e9e1d739e45130f3bf35b6428c9485a01762eccdd4c50d3fb139e46addc52860: Status 404 returned error can't find the container with id e9e1d739e45130f3bf35b6428c9485a01762eccdd4c50d3fb139e46addc52860
Jul 29 00:11:18.793741 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:18.793708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4" event={"ID":"b85541bc-24b6-4645-9f43-7b150b724c1e","Type":"ContainerStarted","Data":"bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a"}
Jul 29 00:11:18.794107 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:18.793752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4" event={"ID":"b85541bc-24b6-4645-9f43-7b150b724c1e","Type":"ContainerStarted","Data":"e9e1d739e45130f3bf35b6428c9485a01762eccdd4c50d3fb139e46addc52860"}
Jul 29 00:11:18.813943 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:18.813895    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4" podStartSLOduration=1.813880538 podStartE2EDuration="1.813880538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:11:18.812217568 +0000 UTC m=+2974.314764100" watchObservedRunningTime="2026-07-29 00:11:18.813880538 +0000 UTC m=+2974.316427100"
Jul 29 00:11:22.039649 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:22.039593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:11:28.019326 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:28.019285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:28.019794 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:28.019362    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:28.032139 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:28.032113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:28.854520 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:28.854493    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:11:52.540431 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.540400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"]
Jul 29 00:11:52.545339 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.545319    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.549894 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.549873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"default-dockercfg-7xbrr\""
Jul 29 00:11:52.550003 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.549892    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"kube-root-ca.crt\""
Jul 29 00:11:52.550003 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.549873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"sub-test-model-fa6130-kserve-self-signed-certs\""
Jul 29 00:11:52.550003 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.549960    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"openshift-service-ca.crt\""
Jul 29 00:11:52.557975 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.557955    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"]
Jul 29 00:11:52.642538 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.642510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-dshm\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.642695 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.642543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354aa4f9-91cd-4300-aa17-db62a597ccee-tls-certs\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.642695 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.642561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-home\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.642695 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.642582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-model-cache\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.642695 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.642679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88xcp\" (UniqueName: \"kubernetes.io/projected/354aa4f9-91cd-4300-aa17-db62a597ccee-kube-api-access-88xcp\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.642836 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.642767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-tmp-dir\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743376 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-model-cache\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743541 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88xcp\" (UniqueName: \"kubernetes.io/projected/354aa4f9-91cd-4300-aa17-db62a597ccee-kube-api-access-88xcp\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743541 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-tmp-dir\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743541 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-dshm\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743711 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354aa4f9-91cd-4300-aa17-db62a597ccee-tls-certs\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743711 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-home\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743851 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-model-cache\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.743995 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-tmp-dir\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.744069 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.743995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-home\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.745926 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.745907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-dshm\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.746020 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.746008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354aa4f9-91cd-4300-aa17-db62a597ccee-tls-certs\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.756865 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.756842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88xcp\" (UniqueName: \"kubernetes.io/projected/354aa4f9-91cd-4300-aa17-db62a597ccee-kube-api-access-88xcp\") pod \"sub-test-model-fa6130-kserve-564f7bf69c-68xp8\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") " pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.854823 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.854747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:11:52.997772 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:52.997747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"]
Jul 29 00:11:52.999473 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:11:52.999446    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod354aa4f9_91cd_4300_aa17_db62a597ccee.slice/crio-38c7cbec91b3bebc82ad38f7cb6e05ae74f932ed4bcc8a2a0a4c6bfc19a14bad WatchSource:0}: Error finding container 38c7cbec91b3bebc82ad38f7cb6e05ae74f932ed4bcc8a2a0a4c6bfc19a14bad: Status 404 returned error can't find the container with id 38c7cbec91b3bebc82ad38f7cb6e05ae74f932ed4bcc8a2a0a4c6bfc19a14bad
Jul 29 00:11:53.964457 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:53.964418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8" event={"ID":"354aa4f9-91cd-4300-aa17-db62a597ccee","Type":"ContainerStarted","Data":"d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84"}
Jul 29 00:11:53.964856 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:53.964465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8" event={"ID":"354aa4f9-91cd-4300-aa17-db62a597ccee","Type":"ContainerStarted","Data":"38c7cbec91b3bebc82ad38f7cb6e05ae74f932ed4bcc8a2a0a4c6bfc19a14bad"}
Jul 29 00:11:53.988727 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:53.988677    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8" podStartSLOduration=1.988662956 podStartE2EDuration="1.988662956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:11:53.986573961 +0000 UTC m=+3009.489120492" watchObservedRunningTime="2026-07-29 00:11:53.988662956 +0000 UTC m=+3009.491209502"
Jul 29 00:11:56.436833 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:11:56.436791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:12:02.855566 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:02.855525    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:12:02.855566 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:02.855579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:12:02.868337 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:02.868310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:12:03.015901 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:03.015867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:12:29.215713 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:29.215675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:12:50.211740 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:50.211662    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:12:56.717479 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:56.717438    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"]
Jul 29 00:12:57.620742 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:12:57.620710    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"]
Jul 29 00:13:09.725500 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:09.725462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:13:10.015556 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.015466    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"]
Jul 29 00:13:10.018880 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.018858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.021689 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.021666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-daedc5\"/\"rate-test-model-daedc5-kserve-self-signed-certs\""
Jul 29 00:13:10.029011 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.028988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"]
Jul 29 00:13:10.069076 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.069046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tmp-dir\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.069076 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.069077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tls-certs\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.069307 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.069096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jtxq\" (UniqueName: \"kubernetes.io/projected/73fe23d4-714e-4b2b-8add-87c6a6c475b8-kube-api-access-2jtxq\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.069307 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.069116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-dshm\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.069307 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.069229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-model-cache\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.069451 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.069335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-home\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170224 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tmp-dir\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170224 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tls-certs\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170480 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jtxq\" (UniqueName: \"kubernetes.io/projected/73fe23d4-714e-4b2b-8add-87c6a6c475b8-kube-api-access-2jtxq\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170480 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-dshm\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170480 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-model-cache\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170671 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-home\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170671 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tmp-dir\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170881 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-home\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.170881 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.170869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-model-cache\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.172653 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.172593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-dshm\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.172872 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.172851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tls-certs\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.179708 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.179688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jtxq\" (UniqueName: \"kubernetes.io/projected/73fe23d4-714e-4b2b-8add-87c6a6c475b8-kube-api-access-2jtxq\") pod \"rate-test-model-daedc5-kserve-6df77f67bc-qrnjn\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") " pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.328395 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.328311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:10.465298 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:10.465269    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"]
Jul 29 00:13:10.466793 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:13:10.466762    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73fe23d4_714e_4b2b_8add_87c6a6c475b8.slice/crio-b4f5ea1f430e08c82fd171a11bc6289c7bdf76fc2d187cd4ee19d606c7d5bfc1 WatchSource:0}: Error finding container b4f5ea1f430e08c82fd171a11bc6289c7bdf76fc2d187cd4ee19d606c7d5bfc1: Status 404 returned error can't find the container with id b4f5ea1f430e08c82fd171a11bc6289c7bdf76fc2d187cd4ee19d606c7d5bfc1
Jul 29 00:13:11.343556 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:11.343517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn" event={"ID":"73fe23d4-714e-4b2b-8add-87c6a6c475b8","Type":"ContainerStarted","Data":"fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8"}
Jul 29 00:13:11.343556 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:11.343564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn" event={"ID":"73fe23d4-714e-4b2b-8add-87c6a6c475b8","Type":"ContainerStarted","Data":"b4f5ea1f430e08c82fd171a11bc6289c7bdf76fc2d187cd4ee19d606c7d5bfc1"}
Jul 29 00:13:11.361521 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:11.361469    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn" podStartSLOduration=2.361453975 podStartE2EDuration="2.361453975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:13:11.360126808 +0000 UTC m=+3086.862673339" watchObservedRunningTime="2026-07-29 00:13:11.361453975 +0000 UTC m=+3086.864000764"
Jul 29 00:13:11.736247 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:11.736203    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4" podUID="b85541bc-24b6-4645-9f43-7b150b724c1e" containerName="main" containerID="cri-o://bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a" gracePeriod=45
Jul 29 00:13:11.998368 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:11.998309    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:13:12.089419 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-tmp-dir\") pod \"b85541bc-24b6-4645-9f43-7b150b724c1e\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") "
Jul 29 00:13:12.089592 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-model-cache\") pod \"b85541bc-24b6-4645-9f43-7b150b724c1e\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") "
Jul 29 00:13:12.089592 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-home\") pod \"b85541bc-24b6-4645-9f43-7b150b724c1e\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") "
Jul 29 00:13:12.089592 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxqk9\" (UniqueName: \"kubernetes.io/projected/b85541bc-24b6-4645-9f43-7b150b724c1e-kube-api-access-sxqk9\") pod \"b85541bc-24b6-4645-9f43-7b150b724c1e\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") "
Jul 29 00:13:12.089592 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85541bc-24b6-4645-9f43-7b150b724c1e-tls-certs\") pod \"b85541bc-24b6-4645-9f43-7b150b724c1e\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") "
Jul 29 00:13:12.089854 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-dshm\") pod \"b85541bc-24b6-4645-9f43-7b150b724c1e\" (UID: \"b85541bc-24b6-4645-9f43-7b150b724c1e\") "
Jul 29 00:13:12.089854 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089713    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b85541bc-24b6-4645-9f43-7b150b724c1e" (UID: "b85541bc-24b6-4645-9f43-7b150b724c1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.089854 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089723    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b85541bc-24b6-4645-9f43-7b150b724c1e" (UID: "b85541bc-24b6-4645-9f43-7b150b724c1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.089854 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-home" (OuterVolumeSpecName: "home") pod "b85541bc-24b6-4645-9f43-7b150b724c1e" (UID: "b85541bc-24b6-4645-9f43-7b150b724c1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.090025 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089932    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.090025 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089947    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.090025 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.089954    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.091801 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.091775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-dshm" (OuterVolumeSpecName: "dshm") pod "b85541bc-24b6-4645-9f43-7b150b724c1e" (UID: "b85541bc-24b6-4645-9f43-7b150b724c1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.091801 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.091776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85541bc-24b6-4645-9f43-7b150b724c1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b85541bc-24b6-4645-9f43-7b150b724c1e" (UID: "b85541bc-24b6-4645-9f43-7b150b724c1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:13:12.091964 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.091872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85541bc-24b6-4645-9f43-7b150b724c1e-kube-api-access-sxqk9" (OuterVolumeSpecName: "kube-api-access-sxqk9") pod "b85541bc-24b6-4645-9f43-7b150b724c1e" (UID: "b85541bc-24b6-4645-9f43-7b150b724c1e"). InnerVolumeSpecName "kube-api-access-sxqk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:13:12.190834 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.190789    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxqk9\" (UniqueName: \"kubernetes.io/projected/b85541bc-24b6-4645-9f43-7b150b724c1e-kube-api-access-sxqk9\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.190834 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.190819    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85541bc-24b6-4645-9f43-7b150b724c1e-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.190834 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.190833    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85541bc-24b6-4645-9f43-7b150b724c1e-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.350746 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.350655    2581 generic.go:356] "Generic (PLEG): container finished" podID="b85541bc-24b6-4645-9f43-7b150b724c1e" containerID="bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a" exitCode=0
Jul 29 00:13:12.350746 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.350732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4" event={"ID":"b85541bc-24b6-4645-9f43-7b150b724c1e","Type":"ContainerDied","Data":"bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a"}
Jul 29 00:13:12.351258 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.350754    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"
Jul 29 00:13:12.351258 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.350780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4" event={"ID":"b85541bc-24b6-4645-9f43-7b150b724c1e","Type":"ContainerDied","Data":"e9e1d739e45130f3bf35b6428c9485a01762eccdd4c50d3fb139e46addc52860"}
Jul 29 00:13:12.351258 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.350802    2581 scope.go:117] "RemoveContainer" containerID="bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a"
Jul 29 00:13:12.359959 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.359945    2581 scope.go:117] "RemoveContainer" containerID="bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a"
Jul 29 00:13:12.360224 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:13:12.360205    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a\": container with ID starting with bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a not found: ID does not exist" containerID="bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a"
Jul 29 00:13:12.360298 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.360233    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a"} err="failed to get container status \"bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a\": rpc error: code = NotFound desc = could not find container \"bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a\": container with ID starting with bf466bb8a2cc7032e0d8721986d2157df22cf6ae7e8b1876555dfce66a68b48a not found: ID does not exist"
Jul 29 00:13:12.371297 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.371274    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"]
Jul 29 00:13:12.375347 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.375326    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/sub-test-model-daedc5-kserve-5945b486dc-4cns4"]
Jul 29 00:13:12.639669 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.639608    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8" podUID="354aa4f9-91cd-4300-aa17-db62a597ccee" containerName="main" containerID="cri-o://d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84" gracePeriod=45
Jul 29 00:13:12.899547 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.899493    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:13:12.982077 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.982047    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85541bc-24b6-4645-9f43-7b150b724c1e" path="/var/lib/kubelet/pods/b85541bc-24b6-4645-9f43-7b150b724c1e/volumes"
Jul 29 00:13:12.997415 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-model-cache\") pod \"354aa4f9-91cd-4300-aa17-db62a597ccee\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") "
Jul 29 00:13:12.997537 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354aa4f9-91cd-4300-aa17-db62a597ccee-tls-certs\") pod \"354aa4f9-91cd-4300-aa17-db62a597ccee\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") "
Jul 29 00:13:12.997537 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-tmp-dir\") pod \"354aa4f9-91cd-4300-aa17-db62a597ccee\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") "
Jul 29 00:13:12.997664 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-dshm\") pod \"354aa4f9-91cd-4300-aa17-db62a597ccee\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") "
Jul 29 00:13:12.997664 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997568    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-home\") pod \"354aa4f9-91cd-4300-aa17-db62a597ccee\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") "
Jul 29 00:13:12.997664 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-model-cache" (OuterVolumeSpecName: "model-cache") pod "354aa4f9-91cd-4300-aa17-db62a597ccee" (UID: "354aa4f9-91cd-4300-aa17-db62a597ccee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.997832 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997673    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88xcp\" (UniqueName: \"kubernetes.io/projected/354aa4f9-91cd-4300-aa17-db62a597ccee-kube-api-access-88xcp\") pod \"354aa4f9-91cd-4300-aa17-db62a597ccee\" (UID: \"354aa4f9-91cd-4300-aa17-db62a597ccee\") "
Jul 29 00:13:12.997832 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997796    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "354aa4f9-91cd-4300-aa17-db62a597ccee" (UID: "354aa4f9-91cd-4300-aa17-db62a597ccee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.997832 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-home" (OuterVolumeSpecName: "home") pod "354aa4f9-91cd-4300-aa17-db62a597ccee" (UID: "354aa4f9-91cd-4300-aa17-db62a597ccee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.998009 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.997993    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.998062 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.998016    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.998062 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.998032    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:12.999667 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.999605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354aa4f9-91cd-4300-aa17-db62a597ccee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "354aa4f9-91cd-4300-aa17-db62a597ccee" (UID: "354aa4f9-91cd-4300-aa17-db62a597ccee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:13:12.999667 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.999652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-dshm" (OuterVolumeSpecName: "dshm") pod "354aa4f9-91cd-4300-aa17-db62a597ccee" (UID: "354aa4f9-91cd-4300-aa17-db62a597ccee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:13:12.999830 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:12.999675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354aa4f9-91cd-4300-aa17-db62a597ccee-kube-api-access-88xcp" (OuterVolumeSpecName: "kube-api-access-88xcp") pod "354aa4f9-91cd-4300-aa17-db62a597ccee" (UID: "354aa4f9-91cd-4300-aa17-db62a597ccee"). InnerVolumeSpecName "kube-api-access-88xcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:13:13.099284 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.099250    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354aa4f9-91cd-4300-aa17-db62a597ccee-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:13.099284 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.099277    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88xcp\" (UniqueName: \"kubernetes.io/projected/354aa4f9-91cd-4300-aa17-db62a597ccee-kube-api-access-88xcp\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:13.099284 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.099289    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354aa4f9-91cd-4300-aa17-db62a597ccee-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:13:13.357894 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.357859    2581 generic.go:356] "Generic (PLEG): container finished" podID="354aa4f9-91cd-4300-aa17-db62a597ccee" containerID="d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84" exitCode=0
Jul 29 00:13:13.358360 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.357914    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"
Jul 29 00:13:13.358360 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.357944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8" event={"ID":"354aa4f9-91cd-4300-aa17-db62a597ccee","Type":"ContainerDied","Data":"d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84"}
Jul 29 00:13:13.358360 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.357985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8" event={"ID":"354aa4f9-91cd-4300-aa17-db62a597ccee","Type":"ContainerDied","Data":"38c7cbec91b3bebc82ad38f7cb6e05ae74f932ed4bcc8a2a0a4c6bfc19a14bad"}
Jul 29 00:13:13.358360 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.358008    2581 scope.go:117] "RemoveContainer" containerID="d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84"
Jul 29 00:13:13.367278 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.367263    2581 scope.go:117] "RemoveContainer" containerID="d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84"
Jul 29 00:13:13.367528 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:13:13.367512    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84\": container with ID starting with d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84 not found: ID does not exist" containerID="d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84"
Jul 29 00:13:13.367581 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.367535    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84"} err="failed to get container status \"d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84\": rpc error: code = NotFound desc = could not find container \"d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84\": container with ID starting with d07e0166d07dfde057a26828eb73baa3d7b9ffaf189edc1524092b276b8f4f84 not found: ID does not exist"
Jul 29 00:13:13.379451 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.379425    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"]
Jul 29 00:13:13.383176 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:13.383155    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/sub-test-model-fa6130-kserve-564f7bf69c-68xp8"]
Jul 29 00:13:14.982249 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:14.982209    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354aa4f9-91cd-4300-aa17-db62a597ccee" path="/var/lib/kubelet/pods/354aa4f9-91cd-4300-aa17-db62a597ccee/volumes"
Jul 29 00:13:19.416856 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:19.416819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:13:20.329335 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:20.329293    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:20.329577 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:20.329356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:20.341919 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:20.341892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:20.404320 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:20.404288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:13:35.123711 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.123671    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"]
Jul 29 00:13:35.126099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.124209    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85541bc-24b6-4645-9f43-7b150b724c1e" containerName="main"
Jul 29 00:13:35.126099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.124228    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85541bc-24b6-4645-9f43-7b150b724c1e" containerName="main"
Jul 29 00:13:35.126099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.124261    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354aa4f9-91cd-4300-aa17-db62a597ccee" containerName="main"
Jul 29 00:13:35.126099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.124270    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="354aa4f9-91cd-4300-aa17-db62a597ccee" containerName="main"
Jul 29 00:13:35.126099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.124363    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85541bc-24b6-4645-9f43-7b150b724c1e" containerName="main"
Jul 29 00:13:35.126099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.124380    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="354aa4f9-91cd-4300-aa17-db62a597ccee" containerName="main"
Jul 29 00:13:35.127281 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.127264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.131276 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.131246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"openshift-service-ca.crt\""
Jul 29 00:13:35.131895 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.131867    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"rate-test-model-fa6130-kserve-self-signed-certs\""
Jul 29 00:13:35.132569 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.132179    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"default-dockercfg-7xbrr\""
Jul 29 00:13:35.132569 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.132309    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fa6130\"/\"kube-root-ca.crt\""
Jul 29 00:13:35.143472 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.143440    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"]
Jul 29 00:13:35.207497 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.207461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnqw\" (UniqueName: \"kubernetes.io/projected/7ef0501b-0221-4ec2-9e68-910f7bc3821a-kube-api-access-drnqw\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.207707 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.207503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-model-cache\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.207707 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.207582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tmp-dir\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.207707 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.207677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-home\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.207828 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.207746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tls-certs\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.207828 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.207819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-dshm\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309044 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.308996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tls-certs\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-dshm\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnqw\" (UniqueName: \"kubernetes.io/projected/7ef0501b-0221-4ec2-9e68-910f7bc3821a-kube-api-access-drnqw\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-model-cache\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tmp-dir\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309239 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-home\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309599 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-model-cache\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309697 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-home\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.309697 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.309651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tmp-dir\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.311828 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.311808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-dshm\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.312112 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.312092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tls-certs\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.318777 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.318756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnqw\" (UniqueName: \"kubernetes.io/projected/7ef0501b-0221-4ec2-9e68-910f7bc3821a-kube-api-access-drnqw\") pod \"rate-test-model-fa6130-kserve-64d6b7b579-q7tvd\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") " pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.443472 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.443386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:35.574886 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:35.574858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"]
Jul 29 00:13:35.576409 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:13:35.576385    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ef0501b_0221_4ec2_9e68_910f7bc3821a.slice/crio-844975f78b86516f99dc03c04dea53f7913769d3dc93a04595834d98e14b57e6 WatchSource:0}: Error finding container 844975f78b86516f99dc03c04dea53f7913769d3dc93a04595834d98e14b57e6: Status 404 returned error can't find the container with id 844975f78b86516f99dc03c04dea53f7913769d3dc93a04595834d98e14b57e6
Jul 29 00:13:36.473063 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:36.473024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd" event={"ID":"7ef0501b-0221-4ec2-9e68-910f7bc3821a","Type":"ContainerStarted","Data":"50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789"}
Jul 29 00:13:36.473063 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:36.473070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd" event={"ID":"7ef0501b-0221-4ec2-9e68-910f7bc3821a","Type":"ContainerStarted","Data":"844975f78b86516f99dc03c04dea53f7913769d3dc93a04595834d98e14b57e6"}
Jul 29 00:13:36.490957 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:36.490906    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd" podStartSLOduration=1.490890193 podStartE2EDuration="1.490890193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:13:36.489333373 +0000 UTC m=+3111.991879899" watchObservedRunningTime="2026-07-29 00:13:36.490890193 +0000 UTC m=+3111.993436752"
Jul 29 00:13:38.816962 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:38.816925    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:13:45.443925 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:45.443890    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:45.444313 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:45.443942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:45.456965 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:45.456939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:13:45.530538 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:13:45.530455    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:14:16.320801 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:14:16.320765    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:14:31.418250 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:14:31.418218    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:14:46.728720 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:14:46.728675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:14:51.309071 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:14:51.309034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"]
Jul 29 00:14:52.304712 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:14:52.304683    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"]
Jul 29 00:15:04.115855 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:04.115816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:15:06.327903 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.327857    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn" podUID="73fe23d4-714e-4b2b-8add-87c6a6c475b8" containerName="main" containerID="cri-o://fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8" gracePeriod=45
Jul 29 00:15:06.573703 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.573678    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:15:06.672099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tmp-dir\") pod \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") "
Jul 29 00:15:06.672273 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jtxq\" (UniqueName: \"kubernetes.io/projected/73fe23d4-714e-4b2b-8add-87c6a6c475b8-kube-api-access-2jtxq\") pod \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") "
Jul 29 00:15:06.672273 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672197    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-dshm\") pod \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") "
Jul 29 00:15:06.672273 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tls-certs\") pod \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") "
Jul 29 00:15:06.672273 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-model-cache\") pod \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") "
Jul 29 00:15:06.672467 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-home\") pod \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\" (UID: \"73fe23d4-714e-4b2b-8add-87c6a6c475b8\") "
Jul 29 00:15:06.672467 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73fe23d4-714e-4b2b-8add-87c6a6c475b8" (UID: "73fe23d4-714e-4b2b-8add-87c6a6c475b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:06.672646 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672597    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:06.672646 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "73fe23d4-714e-4b2b-8add-87c6a6c475b8" (UID: "73fe23d4-714e-4b2b-8add-87c6a6c475b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:06.672646 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.672617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-home" (OuterVolumeSpecName: "home") pod "73fe23d4-714e-4b2b-8add-87c6a6c475b8" (UID: "73fe23d4-714e-4b2b-8add-87c6a6c475b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:06.674445 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.674417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-dshm" (OuterVolumeSpecName: "dshm") pod "73fe23d4-714e-4b2b-8add-87c6a6c475b8" (UID: "73fe23d4-714e-4b2b-8add-87c6a6c475b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:06.674539 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.674442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fe23d4-714e-4b2b-8add-87c6a6c475b8-kube-api-access-2jtxq" (OuterVolumeSpecName: "kube-api-access-2jtxq") pod "73fe23d4-714e-4b2b-8add-87c6a6c475b8" (UID: "73fe23d4-714e-4b2b-8add-87c6a6c475b8"). InnerVolumeSpecName "kube-api-access-2jtxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:15:06.674539 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.674498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73fe23d4-714e-4b2b-8add-87c6a6c475b8" (UID: "73fe23d4-714e-4b2b-8add-87c6a6c475b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:15:06.773792 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.773757    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:06.773792 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.773784    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fe23d4-714e-4b2b-8add-87c6a6c475b8-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:06.773792 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.773795    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:06.773792 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.773802    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fe23d4-714e-4b2b-8add-87c6a6c475b8-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:06.774055 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.773813    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jtxq\" (UniqueName: \"kubernetes.io/projected/73fe23d4-714e-4b2b-8add-87c6a6c475b8-kube-api-access-2jtxq\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:06.908386 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.908350    2581 generic.go:356] "Generic (PLEG): container finished" podID="73fe23d4-714e-4b2b-8add-87c6a6c475b8" containerID="fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8" exitCode=0
Jul 29 00:15:06.908595 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.908394    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"
Jul 29 00:15:06.908595 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.908442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn" event={"ID":"73fe23d4-714e-4b2b-8add-87c6a6c475b8","Type":"ContainerDied","Data":"fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8"}
Jul 29 00:15:06.908595 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.908494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn" event={"ID":"73fe23d4-714e-4b2b-8add-87c6a6c475b8","Type":"ContainerDied","Data":"b4f5ea1f430e08c82fd171a11bc6289c7bdf76fc2d187cd4ee19d606c7d5bfc1"}
Jul 29 00:15:06.908595 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.908518    2581 scope.go:117] "RemoveContainer" containerID="fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8"
Jul 29 00:15:06.917681 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.917661    2581 scope.go:117] "RemoveContainer" containerID="fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8"
Jul 29 00:15:06.917933 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:15:06.917911    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8\": container with ID starting with fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8 not found: ID does not exist" containerID="fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8"
Jul 29 00:15:06.917984 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.917946    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8"} err="failed to get container status \"fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8\": rpc error: code = NotFound desc = could not find container \"fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8\": container with ID starting with fc858ff84e966099f29557a4aa747b3a542e02ab9addcf72940020ef3750c9b8 not found: ID does not exist"
Jul 29 00:15:06.930026 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.929968    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"]
Jul 29 00:15:06.934674 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.934653    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-daedc5/rate-test-model-daedc5-kserve-6df77f67bc-qrnjn"]
Jul 29 00:15:06.982349 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:06.982316    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73fe23d4-714e-4b2b-8add-87c6a6c475b8" path="/var/lib/kubelet/pods/73fe23d4-714e-4b2b-8add-87c6a6c475b8/volumes"
Jul 29 00:15:07.322959 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.322871    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd" podUID="7ef0501b-0221-4ec2-9e68-910f7bc3821a" containerName="main" containerID="cri-o://50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789" gracePeriod=45
Jul 29 00:15:07.577759 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.577691    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:15:07.683571 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.683539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-model-cache\") pod \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") "
Jul 29 00:15:07.683571 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.683578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drnqw\" (UniqueName: \"kubernetes.io/projected/7ef0501b-0221-4ec2-9e68-910f7bc3821a-kube-api-access-drnqw\") pod \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") "
Jul 29 00:15:07.683820 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.683673    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tmp-dir\") pod \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") "
Jul 29 00:15:07.683820 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.683700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-home\") pod \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") "
Jul 29 00:15:07.683820 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.683742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tls-certs\") pod \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") "
Jul 29 00:15:07.683820 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.683769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-dshm\") pod \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\" (UID: \"7ef0501b-0221-4ec2-9e68-910f7bc3821a\") "
Jul 29 00:15:07.684032 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.683864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ef0501b-0221-4ec2-9e68-910f7bc3821a" (UID: "7ef0501b-0221-4ec2-9e68-910f7bc3821a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:07.684032 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.684016    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ef0501b-0221-4ec2-9e68-910f7bc3821a" (UID: "7ef0501b-0221-4ec2-9e68-910f7bc3821a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:07.684144 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.684026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-home" (OuterVolumeSpecName: "home") pod "7ef0501b-0221-4ec2-9e68-910f7bc3821a" (UID: "7ef0501b-0221-4ec2-9e68-910f7bc3821a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:07.684144 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.684134    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:07.684221 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.684157    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:07.684221 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.684174    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:07.685803 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.685779    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef0501b-0221-4ec2-9e68-910f7bc3821a-kube-api-access-drnqw" (OuterVolumeSpecName: "kube-api-access-drnqw") pod "7ef0501b-0221-4ec2-9e68-910f7bc3821a" (UID: "7ef0501b-0221-4ec2-9e68-910f7bc3821a"). InnerVolumeSpecName "kube-api-access-drnqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:15:07.685866 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.685784    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-dshm" (OuterVolumeSpecName: "dshm") pod "7ef0501b-0221-4ec2-9e68-910f7bc3821a" (UID: "7ef0501b-0221-4ec2-9e68-910f7bc3821a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:15:07.685914 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.685884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ef0501b-0221-4ec2-9e68-910f7bc3821a" (UID: "7ef0501b-0221-4ec2-9e68-910f7bc3821a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:15:07.785193 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.785151    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drnqw\" (UniqueName: \"kubernetes.io/projected/7ef0501b-0221-4ec2-9e68-910f7bc3821a-kube-api-access-drnqw\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:07.785193 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.785187    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef0501b-0221-4ec2-9e68-910f7bc3821a-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:07.785193 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.785197    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef0501b-0221-4ec2-9e68-910f7bc3821a-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:15:07.915644 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.915593    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ef0501b-0221-4ec2-9e68-910f7bc3821a" containerID="50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789" exitCode=0
Jul 29 00:15:07.915834 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.915658    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"
Jul 29 00:15:07.915834 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.915662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd" event={"ID":"7ef0501b-0221-4ec2-9e68-910f7bc3821a","Type":"ContainerDied","Data":"50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789"}
Jul 29 00:15:07.915834 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.915718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd" event={"ID":"7ef0501b-0221-4ec2-9e68-910f7bc3821a","Type":"ContainerDied","Data":"844975f78b86516f99dc03c04dea53f7913769d3dc93a04595834d98e14b57e6"}
Jul 29 00:15:07.915834 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.915736    2581 scope.go:117] "RemoveContainer" containerID="50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789"
Jul 29 00:15:07.925054 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.925040    2581 scope.go:117] "RemoveContainer" containerID="50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789"
Jul 29 00:15:07.925313 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:15:07.925295    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789\": container with ID starting with 50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789 not found: ID does not exist" containerID="50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789"
Jul 29 00:15:07.925358 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.925320    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789"} err="failed to get container status \"50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789\": rpc error: code = NotFound desc = could not find container \"50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789\": container with ID starting with 50e200973f9802a5dce4a50d623e9898b6ada611c81048625141759c3a880789 not found: ID does not exist"
Jul 29 00:15:07.937503 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.937481    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"]
Jul 29 00:15:07.941296 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:07.941277    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fa6130/rate-test-model-fa6130-kserve-64d6b7b579-q7tvd"]
Jul 29 00:15:08.982968 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:08.982935    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef0501b-0221-4ec2-9e68-910f7bc3821a" path="/var/lib/kubelet/pods/7ef0501b-0221-4ec2-9e68-910f7bc3821a/volumes"
Jul 29 00:15:16.400950 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.400912    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"]
Jul 29 00:15:16.401348 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.401285    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef0501b-0221-4ec2-9e68-910f7bc3821a" containerName="main"
Jul 29 00:15:16.401348 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.401296    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef0501b-0221-4ec2-9e68-910f7bc3821a" containerName="main"
Jul 29 00:15:16.401348 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.401311    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73fe23d4-714e-4b2b-8add-87c6a6c475b8" containerName="main"
Jul 29 00:15:16.401348 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.401316    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73fe23d4-714e-4b2b-8add-87c6a6c475b8" containerName="main"
Jul 29 00:15:16.401483 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.401380    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef0501b-0221-4ec2-9e68-910f7bc3821a" containerName="main"
Jul 29 00:15:16.401483 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.401393    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73fe23d4-714e-4b2b-8add-87c6a6c475b8" containerName="main"
Jul 29 00:15:16.405934 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.405913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.409284 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.409261    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f318ab-dockercfg-vfvd9\""
Jul 29 00:15:16.410036 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.410016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f318ab\""
Jul 29 00:15:16.422608 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.422573    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"]
Jul 29 00:15:16.427055 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.427028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"]
Jul 29 00:15:16.427193 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.427130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.439930 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.439894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"]
Jul 29 00:15:16.469535 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.469503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqx5q\" (UniqueName: \"kubernetes.io/projected/f0e07381-904e-4847-b3a5-3ce4d0cbba47-kube-api-access-hqx5q\") pod \"payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.469535 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.469539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0f134f-3895-40ea-949f-b49ce8b7020d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.469760 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.469561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e07381-904e-4847-b3a5-3ce4d0cbba47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.469760 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.469700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmn8\" (UniqueName: \"kubernetes.io/projected/5f0f134f-3895-40ea-949f-b49ce8b7020d-kube-api-access-bqmn8\") pod \"payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.570895 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.570855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmn8\" (UniqueName: \"kubernetes.io/projected/5f0f134f-3895-40ea-949f-b49ce8b7020d-kube-api-access-bqmn8\") pod \"payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.571066 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.571007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqx5q\" (UniqueName: \"kubernetes.io/projected/f0e07381-904e-4847-b3a5-3ce4d0cbba47-kube-api-access-hqx5q\") pod \"payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.571066 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.571043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0f134f-3895-40ea-949f-b49ce8b7020d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.571145 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.571068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e07381-904e-4847-b3a5-3ce4d0cbba47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.572026 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.572006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0f134f-3895-40ea-949f-b49ce8b7020d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.572085 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.572006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e07381-904e-4847-b3a5-3ce4d0cbba47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.579166 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.579142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqx5q\" (UniqueName: \"kubernetes.io/projected/f0e07381-904e-4847-b3a5-3ce4d0cbba47-kube-api-access-hqx5q\") pod \"payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.579292 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.579252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmn8\" (UniqueName: \"kubernetes.io/projected/5f0f134f-3895-40ea-949f-b49ce8b7020d-kube-api-access-bqmn8\") pod \"payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.726594 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.726496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.738085 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.737981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:16.888368 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.888337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"]
Jul 29 00:15:16.889341 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:15:16.889316    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0f134f_3895_40ea_949f_b49ce8b7020d.slice/crio-3cbdf5e50810fdba81b3cccbb203f8ecd0831f982ce9172063ee35d03c02c6ad WatchSource:0}: Error finding container 3cbdf5e50810fdba81b3cccbb203f8ecd0831f982ce9172063ee35d03c02c6ad: Status 404 returned error can't find the container with id 3cbdf5e50810fdba81b3cccbb203f8ecd0831f982ce9172063ee35d03c02c6ad
Jul 29 00:15:16.962853 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.962827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn" event={"ID":"5f0f134f-3895-40ea-949f-b49ce8b7020d","Type":"ContainerStarted","Data":"8f8437138d9b0ca318129fa8ca7c05103bd35bd5718700f3aefc620128f3e785"}
Jul 29 00:15:16.962995 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.962877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn" event={"ID":"5f0f134f-3895-40ea-949f-b49ce8b7020d","Type":"ContainerStarted","Data":"3cbdf5e50810fdba81b3cccbb203f8ecd0831f982ce9172063ee35d03c02c6ad"}
Jul 29 00:15:16.962995 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.962904    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:16.993784 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:16.993693    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn" podStartSLOduration=0.993675609 podStartE2EDuration="993.675609ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:15:16.993018328 +0000 UTC m=+3212.495564860" watchObservedRunningTime="2026-07-29 00:15:16.993675609 +0000 UTC m=+3212.496222131"
Jul 29 00:15:17.098492 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:17.098466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"]
Jul 29 00:15:17.100266 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:15:17.100238    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e07381_904e_4847_b3a5_3ce4d0cbba47.slice/crio-0c6a2969b20145675653d679f210692464280822b4bf8cfcfbfadd5060e54f40 WatchSource:0}: Error finding container 0c6a2969b20145675653d679f210692464280822b4bf8cfcfbfadd5060e54f40: Status 404 returned error can't find the container with id 0c6a2969b20145675653d679f210692464280822b4bf8cfcfbfadd5060e54f40
Jul 29 00:15:17.969099 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:17.969052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9" event={"ID":"f0e07381-904e-4847-b3a5-3ce4d0cbba47","Type":"ContainerStarted","Data":"27cb02faeb7ef60bdd1c69ba5beb93af370fcd07d0b6fd292d658922d1a899c2"}
Jul 29 00:15:17.969603 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:17.969114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9" event={"ID":"f0e07381-904e-4847-b3a5-3ce4d0cbba47","Type":"ContainerStarted","Data":"0c6a2969b20145675653d679f210692464280822b4bf8cfcfbfadd5060e54f40"}
Jul 29 00:15:17.969603 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:17.969313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:17.988483 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:17.988430    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9" podStartSLOduration=1.988411654 podStartE2EDuration="1.988411654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:15:17.986854862 +0000 UTC m=+3213.489401395" watchObservedRunningTime="2026-07-29 00:15:17.988411654 +0000 UTC m=+3213.490958190"
Jul 29 00:15:24.025704 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:24.025666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:15:27.971558 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:27.971518    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:15:28.982074 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:28.982043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:15:45.247238 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.247162    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"]
Jul 29 00:15:45.250584 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.250560    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.258966 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.258941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eab07a\""
Jul 29 00:15:45.259109 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.258980    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eab07a-dockercfg-jn2qt\""
Jul 29 00:15:45.278901 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.278871    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"]
Jul 29 00:15:45.299209 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.296109    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"]
Jul 29 00:15:45.301309 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.301283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.325652 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.325595    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"]
Jul 29 00:15:45.334421 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.334392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb25f\" (UniqueName: \"kubernetes.io/projected/6947d6b8-e041-4969-b400-6c365d2d46c3-kube-api-access-sb25f\") pod \"payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.334552 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.334442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6947d6b8-e041-4969-b400-6c365d2d46c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.435835 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.435796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.435835 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.435841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5kc\" (UniqueName: \"kubernetes.io/projected/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-kube-api-access-kt5kc\") pod \"payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.436080 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.435942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb25f\" (UniqueName: \"kubernetes.io/projected/6947d6b8-e041-4969-b400-6c365d2d46c3-kube-api-access-sb25f\") pod \"payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.436080 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.436007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6947d6b8-e041-4969-b400-6c365d2d46c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.436790 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.436767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6947d6b8-e041-4969-b400-6c365d2d46c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.445124 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.445100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb25f\" (UniqueName: \"kubernetes.io/projected/6947d6b8-e041-4969-b400-6c365d2d46c3-kube-api-access-sb25f\") pod \"payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.536613 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.536525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.536613 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.536561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5kc\" (UniqueName: \"kubernetes.io/projected/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-kube-api-access-kt5kc\") pod \"payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.537274 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.537253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.545575 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.545545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5kc\" (UniqueName: \"kubernetes.io/projected/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-kube-api-access-kt5kc\") pod \"payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.560541 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.560511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:45.615813 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.615779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:45.909349 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.909320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"]
Jul 29 00:15:45.910817 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:15:45.910793    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6947d6b8_e041_4969_b400_6c365d2d46c3.slice/crio-ddfbd804d1251b24720e066a3c04c39e7c7c5a8a070200833cb2c8a733ed76a8 WatchSource:0}: Error finding container ddfbd804d1251b24720e066a3c04c39e7c7c5a8a070200833cb2c8a733ed76a8: Status 404 returned error can't find the container with id ddfbd804d1251b24720e066a3c04c39e7c7c5a8a070200833cb2c8a733ed76a8
Jul 29 00:15:45.965708 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:45.965679    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"]
Jul 29 00:15:45.967087 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:15:45.967064    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd08e21e_f7fe_44f9_8682_fa6cb395c1a1.slice/crio-5fd7601e9f821fc87956584442a7f28c40e6aac5f830c9b3ac8094738b6dd5f9 WatchSource:0}: Error finding container 5fd7601e9f821fc87956584442a7f28c40e6aac5f830c9b3ac8094738b6dd5f9: Status 404 returned error can't find the container with id 5fd7601e9f821fc87956584442a7f28c40e6aac5f830c9b3ac8094738b6dd5f9
Jul 29 00:15:46.103122 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.103078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t" event={"ID":"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1","Type":"ContainerStarted","Data":"900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5"}
Jul 29 00:15:46.103327 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.103150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t" event={"ID":"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1","Type":"ContainerStarted","Data":"5fd7601e9f821fc87956584442a7f28c40e6aac5f830c9b3ac8094738b6dd5f9"}
Jul 29 00:15:46.103327 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.103185    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:46.104685 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.104653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8" event={"ID":"6947d6b8-e041-4969-b400-6c365d2d46c3","Type":"ContainerStarted","Data":"5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54"}
Jul 29 00:15:46.104813 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.104701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8" event={"ID":"6947d6b8-e041-4969-b400-6c365d2d46c3","Type":"ContainerStarted","Data":"ddfbd804d1251b24720e066a3c04c39e7c7c5a8a070200833cb2c8a733ed76a8"}
Jul 29 00:15:46.104813 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.104729    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:15:46.122599 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.122554    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t" podStartSLOduration=1.122541035 podStartE2EDuration="1.122541035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:15:46.120979959 +0000 UTC m=+3241.623526491" watchObservedRunningTime="2026-07-29 00:15:46.122541035 +0000 UTC m=+3241.625087563"
Jul 29 00:15:46.140934 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:46.140887    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8" podStartSLOduration=1.140872419 podStartE2EDuration="1.140872419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:15:46.139604219 +0000 UTC m=+3241.642150751" watchObservedRunningTime="2026-07-29 00:15:46.140872419 +0000 UTC m=+3241.643418949"
Jul 29 00:15:49.222327 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:49.222291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:15:57.111350 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:57.111320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:15:57.111843 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:15:57.111389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:16:16.369210 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:16.369173    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t_fd08e21e-f7fe-44f9-8682-fa6cb395c1a1/payload-processing/0.log"
Jul 29 00:16:18.919021 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:18.918989    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"]
Jul 29 00:16:18.922648 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:18.922617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:18.926314 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:18.926292    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f318ab\"/\"ipp-route-f318ab-kserve-self-signed-certs\""
Jul 29 00:16:18.926455 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:18.926314    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f318ab\"/\"default-dockercfg-2m62p\""
Jul 29 00:16:18.926455 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:18.926312    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f318ab\"/\"kube-root-ca.crt\""
Jul 29 00:16:18.926455 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:18.926335    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f318ab\"/\"openshift-service-ca.crt\""
Jul 29 00:16:18.930538 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:18.930512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"]
Jul 29 00:16:19.068838 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.068807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c24a3b7-9e63-4133-97c9-242513b40d17-tls-certs\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.069030 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.068854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-home\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.069030 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.068903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bld4d\" (UniqueName: \"kubernetes.io/projected/0c24a3b7-9e63-4133-97c9-242513b40d17-kube-api-access-bld4d\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.069030 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.068932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-tmp-dir\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.069030 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.068954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-dshm\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.069188 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.069042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-model-cache\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.169833 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.169754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bld4d\" (UniqueName: \"kubernetes.io/projected/0c24a3b7-9e63-4133-97c9-242513b40d17-kube-api-access-bld4d\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.169833 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.169796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-tmp-dir\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.169833 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.169819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-dshm\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.170103 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.169860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-model-cache\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.170103 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.169908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c24a3b7-9e63-4133-97c9-242513b40d17-tls-certs\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.170103 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.170052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-home\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.170313 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.170291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-tmp-dir\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.170373 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.170313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-model-cache\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.170411 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.170389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-home\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.172184 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.172164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-dshm\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.172342 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.172323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c24a3b7-9e63-4133-97c9-242513b40d17-tls-certs\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.177325 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.177303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bld4d\" (UniqueName: \"kubernetes.io/projected/0c24a3b7-9e63-4133-97c9-242513b40d17-kube-api-access-bld4d\") pod \"ipp-route-f318ab-kserve-b66f764b7-q9fvn\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") " pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.234208 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.234169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:19.386694 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:19.386669    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"]
Jul 29 00:16:19.387937 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:16:19.387904    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c24a3b7_9e63_4133_97c9_242513b40d17.slice/crio-e5b71f1f86d0ff0c59b25ad84ca842d34720dc6fc1d684b28a8b5b9edb63dcdc WatchSource:0}: Error finding container e5b71f1f86d0ff0c59b25ad84ca842d34720dc6fc1d684b28a8b5b9edb63dcdc: Status 404 returned error can't find the container with id e5b71f1f86d0ff0c59b25ad84ca842d34720dc6fc1d684b28a8b5b9edb63dcdc
Jul 29 00:16:20.270721 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:20.270683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn" event={"ID":"0c24a3b7-9e63-4133-97c9-242513b40d17","Type":"ContainerStarted","Data":"aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368"}
Jul 29 00:16:20.271209 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:20.270732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn" event={"ID":"0c24a3b7-9e63-4133-97c9-242513b40d17","Type":"ContainerStarted","Data":"e5b71f1f86d0ff0c59b25ad84ca842d34720dc6fc1d684b28a8b5b9edb63dcdc"}
Jul 29 00:16:20.287445 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:20.287398    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn" podStartSLOduration=2.287385344 podStartE2EDuration="2.287385344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:16:20.285823187 +0000 UTC m=+3275.788369719" watchObservedRunningTime="2026-07-29 00:16:20.287385344 +0000 UTC m=+3275.789931912"
Jul 29 00:16:22.620747 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:22.620714    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:16:29.235134 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:29.235095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:29.235134 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:29.235145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:29.247831 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:29.247801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:29.328058 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:29.328027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:16:51.715770 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:16:51.715730    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:17:03.251462 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:03.251434    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9_f0e07381-904e-4847-b3a5-3ce4d0cbba47/payload-processing/0.log"
Jul 29 00:17:03.493705 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:03.493679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t_fd08e21e-f7fe-44f9-8682-fa6cb395c1a1/payload-processing/0.log"
Jul 29 00:17:07.904120 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.904040    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"]
Jul 29 00:17:07.907918 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.907899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:07.910374 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.910353    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e26439\""
Jul 29 00:17:07.910493 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.910432    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e26439-dockercfg-f8lmc\""
Jul 29 00:17:07.918052 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.918026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"]
Jul 29 00:17:07.940526 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.940489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eacfab7-d6cd-4222-b2ec-3af905ffd018-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:07.940678 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.940569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g77xg\" (UniqueName: \"kubernetes.io/projected/9eacfab7-d6cd-4222-b2ec-3af905ffd018-kube-api-access-g77xg\") pod \"payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:07.941145 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.941128    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"]
Jul 29 00:17:07.944505 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.944488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:07.953097 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:07.953068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"]
Jul 29 00:17:08.041792 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.041760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmdzb\" (UniqueName: \"kubernetes.io/projected/cb603b92-d906-4547-9dc0-a6a9824af79c-kube-api-access-vmdzb\") pod \"payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:08.041964 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.041808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g77xg\" (UniqueName: \"kubernetes.io/projected/9eacfab7-d6cd-4222-b2ec-3af905ffd018-kube-api-access-g77xg\") pod \"payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:08.041964 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.041859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb603b92-d906-4547-9dc0-a6a9824af79c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:08.041964 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.041951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eacfab7-d6cd-4222-b2ec-3af905ffd018-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:08.042681 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.042652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eacfab7-d6cd-4222-b2ec-3af905ffd018-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:08.052015 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.051991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g77xg\" (UniqueName: \"kubernetes.io/projected/9eacfab7-d6cd-4222-b2ec-3af905ffd018-kube-api-access-g77xg\") pod \"payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:08.142508 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.142470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmdzb\" (UniqueName: \"kubernetes.io/projected/cb603b92-d906-4547-9dc0-a6a9824af79c-kube-api-access-vmdzb\") pod \"payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:08.142710 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.142528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb603b92-d906-4547-9dc0-a6a9824af79c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:08.143271 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.143250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb603b92-d906-4547-9dc0-a6a9824af79c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:08.152416 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.152393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmdzb\" (UniqueName: \"kubernetes.io/projected/cb603b92-d906-4547-9dc0-a6a9824af79c-kube-api-access-vmdzb\") pod \"payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:08.218895 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.218810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:08.255773 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.255745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:08.572519 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.572495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"]
Jul 29 00:17:08.574006 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:17:08.573982    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eacfab7_d6cd_4222_b2ec_3af905ffd018.slice/crio-c98b8a4c4bdf4482d48b0bbd7f54def2ef444d99c3bcbf8c31b5ccf958d825c5 WatchSource:0}: Error finding container c98b8a4c4bdf4482d48b0bbd7f54def2ef444d99c3bcbf8c31b5ccf958d825c5: Status 404 returned error can't find the container with id c98b8a4c4bdf4482d48b0bbd7f54def2ef444d99c3bcbf8c31b5ccf958d825c5
Jul 29 00:17:08.610179 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:08.610155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"]
Jul 29 00:17:08.612180 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:17:08.612150    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb603b92_d906_4547_9dc0_a6a9824af79c.slice/crio-3339e7da65ee9210c4b76a56ca733517050d6c3b847bd0e00a46c8dec1d20039 WatchSource:0}: Error finding container 3339e7da65ee9210c4b76a56ca733517050d6c3b847bd0e00a46c8dec1d20039: Status 404 returned error can't find the container with id 3339e7da65ee9210c4b76a56ca733517050d6c3b847bd0e00a46c8dec1d20039
Jul 29 00:17:09.515324 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.515285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv" event={"ID":"9eacfab7-d6cd-4222-b2ec-3af905ffd018","Type":"ContainerStarted","Data":"2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432"}
Jul 29 00:17:09.515903 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.515349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv" event={"ID":"9eacfab7-d6cd-4222-b2ec-3af905ffd018","Type":"ContainerStarted","Data":"c98b8a4c4bdf4482d48b0bbd7f54def2ef444d99c3bcbf8c31b5ccf958d825c5"}
Jul 29 00:17:09.515903 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.515572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:09.516961 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.516926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh" event={"ID":"cb603b92-d906-4547-9dc0-a6a9824af79c","Type":"ContainerStarted","Data":"5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b"}
Jul 29 00:17:09.517079 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.516983    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:09.517079 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.516999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh" event={"ID":"cb603b92-d906-4547-9dc0-a6a9824af79c","Type":"ContainerStarted","Data":"3339e7da65ee9210c4b76a56ca733517050d6c3b847bd0e00a46c8dec1d20039"}
Jul 29 00:17:09.534527 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.534483    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv" podStartSLOduration=2.534469228 podStartE2EDuration="2.534469228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:17:09.532452556 +0000 UTC m=+3325.034999087" watchObservedRunningTime="2026-07-29 00:17:09.534469228 +0000 UTC m=+3325.037015759"
Jul 29 00:17:09.549474 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:09.549429    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh" podStartSLOduration=2.54941409 podStartE2EDuration="2.54941409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:17:09.548416537 +0000 UTC m=+3325.050963073" watchObservedRunningTime="2026-07-29 00:17:09.54941409 +0000 UTC m=+3325.051960623"
Jul 29 00:17:11.318907 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:11.318870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:17:20.523914 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:20.523853    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv" podUID="9eacfab7-d6cd-4222-b2ec-3af905ffd018" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.70:9004: connect: connection refused"
Jul 29 00:17:20.524401 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:20.524320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:30.522466 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:30.522435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:37.820491 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:37.820455    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"]
Jul 29 00:17:37.821050 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:37.820704    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh" podUID="cb603b92-d906-4547-9dc0-a6a9824af79c" containerName="payload-processing" containerID="cri-o://5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b" gracePeriod=30
Jul 29 00:17:37.829193 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:37.829159    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"]
Jul 29 00:17:37.829486 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:37.829433    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv" podUID="9eacfab7-d6cd-4222-b2ec-3af905ffd018" containerName="payload-pre-processing" containerID="cri-o://2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432" gracePeriod=30
Jul 29 00:17:38.114866 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.114839    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:38.117601 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.117585    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:38.247266 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.247234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g77xg\" (UniqueName: \"kubernetes.io/projected/9eacfab7-d6cd-4222-b2ec-3af905ffd018-kube-api-access-g77xg\") pod \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") "
Jul 29 00:17:38.247266 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.247275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmdzb\" (UniqueName: \"kubernetes.io/projected/cb603b92-d906-4547-9dc0-a6a9824af79c-kube-api-access-vmdzb\") pod \"cb603b92-d906-4547-9dc0-a6a9824af79c\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") "
Jul 29 00:17:38.247520 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.247315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb603b92-d906-4547-9dc0-a6a9824af79c-plugins-config-volume\") pod \"cb603b92-d906-4547-9dc0-a6a9824af79c\" (UID: \"cb603b92-d906-4547-9dc0-a6a9824af79c\") "
Jul 29 00:17:38.247520 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.247398    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eacfab7-d6cd-4222-b2ec-3af905ffd018-plugins-config-volume\") pod \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\" (UID: \"9eacfab7-d6cd-4222-b2ec-3af905ffd018\") "
Jul 29 00:17:38.247833 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.247799    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb603b92-d906-4547-9dc0-a6a9824af79c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb603b92-d906-4547-9dc0-a6a9824af79c" (UID: "cb603b92-d906-4547-9dc0-a6a9824af79c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:17:38.247833 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.247813    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eacfab7-d6cd-4222-b2ec-3af905ffd018-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9eacfab7-d6cd-4222-b2ec-3af905ffd018" (UID: "9eacfab7-d6cd-4222-b2ec-3af905ffd018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:17:38.249494 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.249467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb603b92-d906-4547-9dc0-a6a9824af79c-kube-api-access-vmdzb" (OuterVolumeSpecName: "kube-api-access-vmdzb") pod "cb603b92-d906-4547-9dc0-a6a9824af79c" (UID: "cb603b92-d906-4547-9dc0-a6a9824af79c"). InnerVolumeSpecName "kube-api-access-vmdzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:17:38.249586 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.249560    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eacfab7-d6cd-4222-b2ec-3af905ffd018-kube-api-access-g77xg" (OuterVolumeSpecName: "kube-api-access-g77xg") pod "9eacfab7-d6cd-4222-b2ec-3af905ffd018" (UID: "9eacfab7-d6cd-4222-b2ec-3af905ffd018"). InnerVolumeSpecName "kube-api-access-g77xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:17:38.348292 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.348206    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g77xg\" (UniqueName: \"kubernetes.io/projected/9eacfab7-d6cd-4222-b2ec-3af905ffd018-kube-api-access-g77xg\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:17:38.348292 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.348237    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmdzb\" (UniqueName: \"kubernetes.io/projected/cb603b92-d906-4547-9dc0-a6a9824af79c-kube-api-access-vmdzb\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:17:38.348292 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.348246    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb603b92-d906-4547-9dc0-a6a9824af79c-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:17:38.348292 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.348255    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eacfab7-d6cd-4222-b2ec-3af905ffd018-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:17:38.675211 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.675178    2581 generic.go:356] "Generic (PLEG): container finished" podID="cb603b92-d906-4547-9dc0-a6a9824af79c" containerID="5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b" exitCode=0
Jul 29 00:17:38.675402 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.675221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"
Jul 29 00:17:38.675402 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.675268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh" event={"ID":"cb603b92-d906-4547-9dc0-a6a9824af79c","Type":"ContainerDied","Data":"5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b"}
Jul 29 00:17:38.675402 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.675319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh" event={"ID":"cb603b92-d906-4547-9dc0-a6a9824af79c","Type":"ContainerDied","Data":"3339e7da65ee9210c4b76a56ca733517050d6c3b847bd0e00a46c8dec1d20039"}
Jul 29 00:17:38.675402 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.675346    2581 scope.go:117] "RemoveContainer" containerID="5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b"
Jul 29 00:17:38.676786 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.676759    2581 generic.go:356] "Generic (PLEG): container finished" podID="9eacfab7-d6cd-4222-b2ec-3af905ffd018" containerID="2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432" exitCode=0
Jul 29 00:17:38.676908 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.676796    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"
Jul 29 00:17:38.676908 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.676826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv" event={"ID":"9eacfab7-d6cd-4222-b2ec-3af905ffd018","Type":"ContainerDied","Data":"2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432"}
Jul 29 00:17:38.676908 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.676872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv" event={"ID":"9eacfab7-d6cd-4222-b2ec-3af905ffd018","Type":"ContainerDied","Data":"c98b8a4c4bdf4482d48b0bbd7f54def2ef444d99c3bcbf8c31b5ccf958d825c5"}
Jul 29 00:17:38.685489 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.685437    2581 scope.go:117] "RemoveContainer" containerID="5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b"
Jul 29 00:17:38.685758 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:17:38.685739    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b\": container with ID starting with 5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b not found: ID does not exist" containerID="5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b"
Jul 29 00:17:38.685846 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.685766    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b"} err="failed to get container status \"5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b\": rpc error: code = NotFound desc = could not find container \"5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b\": container with ID starting with 5ce406c8c0a5b1471cc727c6660af87a9128f2e939a01e498da707e9a09eb69b not found: ID does not exist"
Jul 29 00:17:38.685846 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.685783    2581 scope.go:117] "RemoveContainer" containerID="2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432"
Jul 29 00:17:38.693609 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.693594    2581 scope.go:117] "RemoveContainer" containerID="2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432"
Jul 29 00:17:38.693864 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:17:38.693847    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432\": container with ID starting with 2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432 not found: ID does not exist" containerID="2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432"
Jul 29 00:17:38.693910 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.693873    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432"} err="failed to get container status \"2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432\": rpc error: code = NotFound desc = could not find container \"2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432\": container with ID starting with 2a4ff7b44ec7260dc3a87944036d220e4ee0a264b20d520d9661e6d2676ed432 not found: ID does not exist"
Jul 29 00:17:38.700513 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.700494    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"]
Jul 29 00:17:38.704993 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.704972    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e26439-547f8958bd-k7fhv"]
Jul 29 00:17:38.715319 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.715298    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"]
Jul 29 00:17:38.719508 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.719489    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e26439-64b564646b-9mdxh"]
Jul 29 00:17:38.983344 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.983269    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eacfab7-d6cd-4222-b2ec-3af905ffd018" path="/var/lib/kubelet/pods/9eacfab7-d6cd-4222-b2ec-3af905ffd018/volumes"
Jul 29 00:17:38.983779 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:38.983678    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb603b92-d906-4547-9dc0-a6a9824af79c" path="/var/lib/kubelet/pods/cb603b92-d906-4547-9dc0-a6a9824af79c/volumes"
Jul 29 00:17:42.214549 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:17:42.214514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:18:02.715030 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:02.714994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:18:03.605751 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.605718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"]
Jul 29 00:18:03.606005 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.605952    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9" podUID="f0e07381-904e-4847-b3a5-3ce4d0cbba47" containerName="payload-processing" containerID="cri-o://27cb02faeb7ef60bdd1c69ba5beb93af370fcd07d0b6fd292d658922d1a899c2" gracePeriod=30
Jul 29 00:18:03.614086 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.614059    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"]
Jul 29 00:18:03.614347 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.614321    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn" podUID="5f0f134f-3895-40ea-949f-b49ce8b7020d" containerName="payload-pre-processing" containerID="cri-o://8f8437138d9b0ca318129fa8ca7c05103bd35bd5718700f3aefc620128f3e785" gracePeriod=30
Jul 29 00:18:03.819086 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.818920    2581 generic.go:356] "Generic (PLEG): container finished" podID="5f0f134f-3895-40ea-949f-b49ce8b7020d" containerID="8f8437138d9b0ca318129fa8ca7c05103bd35bd5718700f3aefc620128f3e785" exitCode=0
Jul 29 00:18:03.819086 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.819079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn" event={"ID":"5f0f134f-3895-40ea-949f-b49ce8b7020d","Type":"ContainerDied","Data":"8f8437138d9b0ca318129fa8ca7c05103bd35bd5718700f3aefc620128f3e785"}
Jul 29 00:18:03.821111 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.821074    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0e07381-904e-4847-b3a5-3ce4d0cbba47" containerID="27cb02faeb7ef60bdd1c69ba5beb93af370fcd07d0b6fd292d658922d1a899c2" exitCode=0
Jul 29 00:18:03.821233 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.821116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9" event={"ID":"f0e07381-904e-4847-b3a5-3ce4d0cbba47","Type":"ContainerDied","Data":"27cb02faeb7ef60bdd1c69ba5beb93af370fcd07d0b6fd292d658922d1a899c2"}
Jul 29 00:18:03.916557 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.916519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:18:03.919507 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.919481    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:18:03.985272 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.985237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e07381-904e-4847-b3a5-3ce4d0cbba47-plugins-config-volume\") pod \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") "
Jul 29 00:18:03.985485 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.985288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqx5q\" (UniqueName: \"kubernetes.io/projected/f0e07381-904e-4847-b3a5-3ce4d0cbba47-kube-api-access-hqx5q\") pod \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\" (UID: \"f0e07381-904e-4847-b3a5-3ce4d0cbba47\") "
Jul 29 00:18:03.985485 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.985330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqmn8\" (UniqueName: \"kubernetes.io/projected/5f0f134f-3895-40ea-949f-b49ce8b7020d-kube-api-access-bqmn8\") pod \"5f0f134f-3895-40ea-949f-b49ce8b7020d\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") "
Jul 29 00:18:03.985485 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.985367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0f134f-3895-40ea-949f-b49ce8b7020d-plugins-config-volume\") pod \"5f0f134f-3895-40ea-949f-b49ce8b7020d\" (UID: \"5f0f134f-3895-40ea-949f-b49ce8b7020d\") "
Jul 29 00:18:03.985760 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.985649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e07381-904e-4847-b3a5-3ce4d0cbba47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0e07381-904e-4847-b3a5-3ce4d0cbba47" (UID: "f0e07381-904e-4847-b3a5-3ce4d0cbba47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:18:03.985860 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.985838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0f134f-3895-40ea-949f-b49ce8b7020d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f0f134f-3895-40ea-949f-b49ce8b7020d" (UID: "5f0f134f-3895-40ea-949f-b49ce8b7020d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:18:03.987487 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.987461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e07381-904e-4847-b3a5-3ce4d0cbba47-kube-api-access-hqx5q" (OuterVolumeSpecName: "kube-api-access-hqx5q") pod "f0e07381-904e-4847-b3a5-3ce4d0cbba47" (UID: "f0e07381-904e-4847-b3a5-3ce4d0cbba47"). InnerVolumeSpecName "kube-api-access-hqx5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:18:03.987572 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:03.987510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0f134f-3895-40ea-949f-b49ce8b7020d-kube-api-access-bqmn8" (OuterVolumeSpecName: "kube-api-access-bqmn8") pod "5f0f134f-3895-40ea-949f-b49ce8b7020d" (UID: "5f0f134f-3895-40ea-949f-b49ce8b7020d"). InnerVolumeSpecName "kube-api-access-bqmn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:18:04.086226 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.086184    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0e07381-904e-4847-b3a5-3ce4d0cbba47-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:04.086226 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.086215    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqx5q\" (UniqueName: \"kubernetes.io/projected/f0e07381-904e-4847-b3a5-3ce4d0cbba47-kube-api-access-hqx5q\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:04.086226 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.086225    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmn8\" (UniqueName: \"kubernetes.io/projected/5f0f134f-3895-40ea-949f-b49ce8b7020d-kube-api-access-bqmn8\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:04.086226 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.086234    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0f134f-3895-40ea-949f-b49ce8b7020d-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:04.827593 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.827566    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"
Jul 29 00:18:04.828061 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.827610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn" event={"ID":"5f0f134f-3895-40ea-949f-b49ce8b7020d","Type":"ContainerDied","Data":"3cbdf5e50810fdba81b3cccbb203f8ecd0831f982ce9172063ee35d03c02c6ad"}
Jul 29 00:18:04.828061 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.827697    2581 scope.go:117] "RemoveContainer" containerID="8f8437138d9b0ca318129fa8ca7c05103bd35bd5718700f3aefc620128f3e785"
Jul 29 00:18:04.829081 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.828917    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"
Jul 29 00:18:04.829081 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.828924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9" event={"ID":"f0e07381-904e-4847-b3a5-3ce4d0cbba47","Type":"ContainerDied","Data":"0c6a2969b20145675653d679f210692464280822b4bf8cfcfbfadd5060e54f40"}
Jul 29 00:18:04.838286 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.838267    2581 scope.go:117] "RemoveContainer" containerID="27cb02faeb7ef60bdd1c69ba5beb93af370fcd07d0b6fd292d658922d1a899c2"
Jul 29 00:18:04.849713 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.849689    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"]
Jul 29 00:18:04.853567 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.853547    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f318ab-59bddb68bd-hvznn"]
Jul 29 00:18:04.863219 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.863189    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"]
Jul 29 00:18:04.869582 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.869557    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f318ab-6f7499c9cb-br2q9"]
Jul 29 00:18:04.982329 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.982252    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0f134f-3895-40ea-949f-b49ce8b7020d" path="/var/lib/kubelet/pods/5f0f134f-3895-40ea-949f-b49ce8b7020d/volumes"
Jul 29 00:18:04.982648 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:04.982618    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e07381-904e-4847-b3a5-3ce4d0cbba47" path="/var/lib/kubelet/pods/f0e07381-904e-4847-b3a5-3ce4d0cbba47/volumes"
Jul 29 00:18:09.120273 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:09.120239    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"]
Jul 29 00:18:13.817029 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:13.816981    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:18:24.139701 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.139659    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn" podUID="0c24a3b7-9e63-4133-97c9-242513b40d17" containerName="main" containerID="cri-o://aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368" gracePeriod=45
Jul 29 00:18:24.396311 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.396255    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:18:24.487647 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.487595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bld4d\" (UniqueName: \"kubernetes.io/projected/0c24a3b7-9e63-4133-97c9-242513b40d17-kube-api-access-bld4d\") pod \"0c24a3b7-9e63-4133-97c9-242513b40d17\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") "
Jul 29 00:18:24.487848 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.487783    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-home\") pod \"0c24a3b7-9e63-4133-97c9-242513b40d17\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") "
Jul 29 00:18:24.487848 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.487816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c24a3b7-9e63-4133-97c9-242513b40d17-tls-certs\") pod \"0c24a3b7-9e63-4133-97c9-242513b40d17\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") "
Jul 29 00:18:24.487967 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.487872    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-tmp-dir\") pod \"0c24a3b7-9e63-4133-97c9-242513b40d17\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") "
Jul 29 00:18:24.487967 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.487897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-model-cache\") pod \"0c24a3b7-9e63-4133-97c9-242513b40d17\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") "
Jul 29 00:18:24.487967 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.487954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-dshm\") pod \"0c24a3b7-9e63-4133-97c9-242513b40d17\" (UID: \"0c24a3b7-9e63-4133-97c9-242513b40d17\") "
Jul 29 00:18:24.488168 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.488138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-home" (OuterVolumeSpecName: "home") pod "0c24a3b7-9e63-4133-97c9-242513b40d17" (UID: "0c24a3b7-9e63-4133-97c9-242513b40d17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:18:24.488330 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.488269    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-home\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:24.488409 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.488336    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c24a3b7-9e63-4133-97c9-242513b40d17" (UID: "0c24a3b7-9e63-4133-97c9-242513b40d17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:18:24.488469 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.488400    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c24a3b7-9e63-4133-97c9-242513b40d17" (UID: "0c24a3b7-9e63-4133-97c9-242513b40d17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:18:24.490096 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.490071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c24a3b7-9e63-4133-97c9-242513b40d17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c24a3b7-9e63-4133-97c9-242513b40d17" (UID: "0c24a3b7-9e63-4133-97c9-242513b40d17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 00:18:24.490096 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.490087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c24a3b7-9e63-4133-97c9-242513b40d17-kube-api-access-bld4d" (OuterVolumeSpecName: "kube-api-access-bld4d") pod "0c24a3b7-9e63-4133-97c9-242513b40d17" (UID: "0c24a3b7-9e63-4133-97c9-242513b40d17"). InnerVolumeSpecName "kube-api-access-bld4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:18:24.490224 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.490090    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-dshm" (OuterVolumeSpecName: "dshm") pod "0c24a3b7-9e63-4133-97c9-242513b40d17" (UID: "0c24a3b7-9e63-4133-97c9-242513b40d17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 00:18:24.589503 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.589469    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c24a3b7-9e63-4133-97c9-242513b40d17-tls-certs\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:24.589503 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.589494    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-tmp-dir\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:24.589503 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.589503    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-model-cache\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:24.589503 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.589511    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c24a3b7-9e63-4133-97c9-242513b40d17-dshm\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:24.589878 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.589519    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bld4d\" (UniqueName: \"kubernetes.io/projected/0c24a3b7-9e63-4133-97c9-242513b40d17-kube-api-access-bld4d\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:18:24.929884 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.929845    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c24a3b7-9e63-4133-97c9-242513b40d17" containerID="aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368" exitCode=0
Jul 29 00:18:24.929884 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.929884    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"
Jul 29 00:18:24.930122 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.929905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn" event={"ID":"0c24a3b7-9e63-4133-97c9-242513b40d17","Type":"ContainerDied","Data":"aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368"}
Jul 29 00:18:24.930122 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.929941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn" event={"ID":"0c24a3b7-9e63-4133-97c9-242513b40d17","Type":"ContainerDied","Data":"e5b71f1f86d0ff0c59b25ad84ca842d34720dc6fc1d684b28a8b5b9edb63dcdc"}
Jul 29 00:18:24.930122 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.929958    2581 scope.go:117] "RemoveContainer" containerID="aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368"
Jul 29 00:18:24.939535 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.939519    2581 scope.go:117] "RemoveContainer" containerID="aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368"
Jul 29 00:18:24.939804 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:18:24.939786    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368\": container with ID starting with aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368 not found: ID does not exist" containerID="aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368"
Jul 29 00:18:24.939869 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.939821    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368"} err="failed to get container status \"aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368\": rpc error: code = NotFound desc = could not find container \"aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368\": container with ID starting with aea3a214d956db64925837a9fe9f5bfcc1f5570184733e235fdcef43516ba368 not found: ID does not exist"
Jul 29 00:18:24.951105 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.951077    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"]
Jul 29 00:18:24.954248 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.954227    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f318ab/ipp-route-f318ab-kserve-b66f764b7-q9fvn"]
Jul 29 00:18:24.982576 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:18:24.982547    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c24a3b7-9e63-4133-97c9-242513b40d17" path="/var/lib/kubelet/pods/0c24a3b7-9e63-4133-97c9-242513b40d17/volumes"
Jul 29 00:19:03.296129 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.296091    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"]
Jul 29 00:19:03.296541 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.296363    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t" podUID="fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" containerName="payload-processing" containerID="cri-o://900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5" gracePeriod=30
Jul 29 00:19:03.300211 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.300184    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"]
Jul 29 00:19:03.300563 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.300535    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8" podUID="6947d6b8-e041-4969-b400-6c365d2d46c3" containerName="payload-pre-processing" containerID="cri-o://5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54" gracePeriod=30
Jul 29 00:19:03.587682 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.587657    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:19:03.590450 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.590427    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:19:03.662050 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.662018    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt5kc\" (UniqueName: \"kubernetes.io/projected/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-kube-api-access-kt5kc\") pod \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") "
Jul 29 00:19:03.662247 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.662086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-plugins-config-volume\") pod \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\" (UID: \"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1\") "
Jul 29 00:19:03.662247 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.662133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb25f\" (UniqueName: \"kubernetes.io/projected/6947d6b8-e041-4969-b400-6c365d2d46c3-kube-api-access-sb25f\") pod \"6947d6b8-e041-4969-b400-6c365d2d46c3\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") "
Jul 29 00:19:03.662247 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.662158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6947d6b8-e041-4969-b400-6c365d2d46c3-plugins-config-volume\") pod \"6947d6b8-e041-4969-b400-6c365d2d46c3\" (UID: \"6947d6b8-e041-4969-b400-6c365d2d46c3\") "
Jul 29 00:19:03.662558 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.662529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" (UID: "fd08e21e-f7fe-44f9-8682-fa6cb395c1a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:19:03.662644 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.662538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6947d6b8-e041-4969-b400-6c365d2d46c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6947d6b8-e041-4969-b400-6c365d2d46c3" (UID: "6947d6b8-e041-4969-b400-6c365d2d46c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:19:03.664313 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.664291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-kube-api-access-kt5kc" (OuterVolumeSpecName: "kube-api-access-kt5kc") pod "fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" (UID: "fd08e21e-f7fe-44f9-8682-fa6cb395c1a1"). InnerVolumeSpecName "kube-api-access-kt5kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:19:03.664409 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.664331    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6947d6b8-e041-4969-b400-6c365d2d46c3-kube-api-access-sb25f" (OuterVolumeSpecName: "kube-api-access-sb25f") pod "6947d6b8-e041-4969-b400-6c365d2d46c3" (UID: "6947d6b8-e041-4969-b400-6c365d2d46c3"). InnerVolumeSpecName "kube-api-access-sb25f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:19:03.762975 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.762940    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt5kc\" (UniqueName: \"kubernetes.io/projected/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-kube-api-access-kt5kc\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:03.762975 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.762967    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:03.762975 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.762978    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb25f\" (UniqueName: \"kubernetes.io/projected/6947d6b8-e041-4969-b400-6c365d2d46c3-kube-api-access-sb25f\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:03.763224 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:03.762988    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6947d6b8-e041-4969-b400-6c365d2d46c3-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:04.115702 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.115662    2581 generic.go:356] "Generic (PLEG): container finished" podID="6947d6b8-e041-4969-b400-6c365d2d46c3" containerID="5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54" exitCode=0
Jul 29 00:19:04.115892 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.115709    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"
Jul 29 00:19:04.115892 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.115745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8" event={"ID":"6947d6b8-e041-4969-b400-6c365d2d46c3","Type":"ContainerDied","Data":"5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54"}
Jul 29 00:19:04.115892 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.115806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8" event={"ID":"6947d6b8-e041-4969-b400-6c365d2d46c3","Type":"ContainerDied","Data":"ddfbd804d1251b24720e066a3c04c39e7c7c5a8a070200833cb2c8a733ed76a8"}
Jul 29 00:19:04.115892 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.115830    2581 scope.go:117] "RemoveContainer" containerID="5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54"
Jul 29 00:19:04.117216 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.117190    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" containerID="900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5" exitCode=0
Jul 29 00:19:04.117344 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.117245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t" event={"ID":"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1","Type":"ContainerDied","Data":"900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5"}
Jul 29 00:19:04.117344 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.117256    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"
Jul 29 00:19:04.117344 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.117290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t" event={"ID":"fd08e21e-f7fe-44f9-8682-fa6cb395c1a1","Type":"ContainerDied","Data":"5fd7601e9f821fc87956584442a7f28c40e6aac5f830c9b3ac8094738b6dd5f9"}
Jul 29 00:19:04.126543 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.126519    2581 scope.go:117] "RemoveContainer" containerID="5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54"
Jul 29 00:19:04.126960 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:19:04.126895    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54\": container with ID starting with 5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54 not found: ID does not exist" containerID="5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54"
Jul 29 00:19:04.126960 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.126934    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54"} err="failed to get container status \"5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54\": rpc error: code = NotFound desc = could not find container \"5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54\": container with ID starting with 5d43b526f60076f61cb1b279376c56552d90a1131772567f318de34171a33f54 not found: ID does not exist"
Jul 29 00:19:04.126960 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.126960    2581 scope.go:117] "RemoveContainer" containerID="900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5"
Jul 29 00:19:04.135262 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.135245    2581 scope.go:117] "RemoveContainer" containerID="900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5"
Jul 29 00:19:04.135545 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:19:04.135527    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5\": container with ID starting with 900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5 not found: ID does not exist" containerID="900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5"
Jul 29 00:19:04.135590 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.135551    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5"} err="failed to get container status \"900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5\": rpc error: code = NotFound desc = could not find container \"900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5\": container with ID starting with 900b4cb7bfc289d87e211acd9ca8b4aa24e6ae3039c0f5c997bad35d2b9a0cf5 not found: ID does not exist"
Jul 29 00:19:04.139565 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.139539    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"]
Jul 29 00:19:04.143647 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.143610    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eab07a-54df54b476-xxmc8"]
Jul 29 00:19:04.154464 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.154425    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"]
Jul 29 00:19:04.159490 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.159462    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eab07a-7848ffb6d8-rbg7t"]
Jul 29 00:19:04.981409 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.981376    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6947d6b8-e041-4969-b400-6c365d2d46c3" path="/var/lib/kubelet/pods/6947d6b8-e041-4969-b400-6c365d2d46c3/volumes"
Jul 29 00:19:04.981827 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:04.981766    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" path="/var/lib/kubelet/pods/fd08e21e-f7fe-44f9-8682-fa6cb395c1a1/volumes"
Jul 29 00:19:06.424947 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:06.424909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:19:20.880000 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:20.879960    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"]
Jul 29 00:19:20.880653 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:20.880217    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l" podUID="bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" containerName="payload-processing" containerID="cri-o://1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33" gracePeriod=30
Jul 29 00:19:20.885931 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:20.885901    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"]
Jul 29 00:19:20.886166 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:20.886119    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4" podUID="33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" containerName="payload-pre-processing" containerID="cri-o://31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79" gracePeriod=30
Jul 29 00:19:21.133341 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.133272    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:19:21.176822 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.176795    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:19:21.203226 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.203190    2581 generic.go:356] "Generic (PLEG): container finished" podID="33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" containerID="31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79" exitCode=0
Jul 29 00:19:21.203403 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.203239    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"
Jul 29 00:19:21.203403 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.203272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4" event={"ID":"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1","Type":"ContainerDied","Data":"31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79"}
Jul 29 00:19:21.203403 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.203316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4" event={"ID":"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1","Type":"ContainerDied","Data":"04e83911ec4fe09c14b30cd798cd479ccb4aad071fa7c9d5843a8af824b2e740"}
Jul 29 00:19:21.203403 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.203348    2581 scope.go:117] "RemoveContainer" containerID="31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79"
Jul 29 00:19:21.204648 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.204609    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" containerID="1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33" exitCode=0
Jul 29 00:19:21.204796 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.204656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l" event={"ID":"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28","Type":"ContainerDied","Data":"1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33"}
Jul 29 00:19:21.204796 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.204673    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"
Jul 29 00:19:21.204796 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.204703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l" event={"ID":"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28","Type":"ContainerDied","Data":"b14125867f34b25f5d43d62cc6218412a1ea96f3a0289790c88c03a50a41766a"}
Jul 29 00:19:21.212049 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.212029    2581 scope.go:117] "RemoveContainer" containerID="31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79"
Jul 29 00:19:21.212357 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:19:21.212336    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79\": container with ID starting with 31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79 not found: ID does not exist" containerID="31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79"
Jul 29 00:19:21.212447 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.212365    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79"} err="failed to get container status \"31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79\": rpc error: code = NotFound desc = could not find container \"31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79\": container with ID starting with 31da713008a2debe925c3f3863309085b09f7e1a93a3b34d02486c943e09cc79 not found: ID does not exist"
Jul 29 00:19:21.212447 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.212382    2581 scope.go:117] "RemoveContainer" containerID="1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33"
Jul 29 00:19:21.222254 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.222232    2581 scope.go:117] "RemoveContainer" containerID="1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33"
Jul 29 00:19:21.222550 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:19:21.222532    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33\": container with ID starting with 1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33 not found: ID does not exist" containerID="1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33"
Jul 29 00:19:21.222599 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.222559    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33"} err="failed to get container status \"1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33\": rpc error: code = NotFound desc = could not find container \"1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33\": container with ID starting with 1181457e8d8439accf98b45584cea5e5f3d5116ad6a13b03b68dd53e9a603e33 not found: ID does not exist"
Jul 29 00:19:21.224858 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.224841    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmcwx\" (UniqueName: \"kubernetes.io/projected/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-kube-api-access-vmcwx\") pod \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") "
Jul 29 00:19:21.224928 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.224907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-plugins-config-volume\") pod \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") "
Jul 29 00:19:21.224968 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.224933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-plugins-config-volume\") pod \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\" (UID: \"33f34dc0-5b9c-4fc5-9f15-83b74a0984a1\") "
Jul 29 00:19:21.224968 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.224964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdj5t\" (UniqueName: \"kubernetes.io/projected/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-kube-api-access-jdj5t\") pod \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\" (UID: \"bf0ddcf9-eb72-4d24-8ce8-7acb01019a28\") "
Jul 29 00:19:21.225296 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.225274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" (UID: "bf0ddcf9-eb72-4d24-8ce8-7acb01019a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:19:21.225379 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.225292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" (UID: "33f34dc0-5b9c-4fc5-9f15-83b74a0984a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:19:21.226899 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.226876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-kube-api-access-vmcwx" (OuterVolumeSpecName: "kube-api-access-vmcwx") pod "33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" (UID: "33f34dc0-5b9c-4fc5-9f15-83b74a0984a1"). InnerVolumeSpecName "kube-api-access-vmcwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:19:21.227081 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.227055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-kube-api-access-jdj5t" (OuterVolumeSpecName: "kube-api-access-jdj5t") pod "bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" (UID: "bf0ddcf9-eb72-4d24-8ce8-7acb01019a28"). InnerVolumeSpecName "kube-api-access-jdj5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:19:21.326264 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.326231    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdj5t\" (UniqueName: \"kubernetes.io/projected/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-kube-api-access-jdj5t\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:21.326264 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.326258    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmcwx\" (UniqueName: \"kubernetes.io/projected/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-kube-api-access-vmcwx\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:21.326264 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.326267    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:21.326505 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.326276    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:21.525633 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.525592    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"]
Jul 29 00:19:21.529739 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.529713    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-daedc5-d6fb69fb-5tzv4"]
Jul 29 00:19:21.540424 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.540397    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"]
Jul 29 00:19:21.544768 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:21.544738    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-daedc5-597f8ddcf4-skg4l"]
Jul 29 00:19:22.982988 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:22.982944    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" path="/var/lib/kubelet/pods/33f34dc0-5b9c-4fc5-9f15-83b74a0984a1/volumes"
Jul 29 00:19:22.983512 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:22.983482    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" path="/var/lib/kubelet/pods/bf0ddcf9-eb72-4d24-8ce8-7acb01019a28/volumes"
Jul 29 00:19:24.015674 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:24.015616    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:19:38.600919 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.600883    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"]
Jul 29 00:19:38.601458 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.601119    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs" podUID="5f6a28c7-0130-4df3-99ef-0679a09f9399" containerName="payload-processing" containerID="cri-o://adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343" gracePeriod=30
Jul 29 00:19:38.615940 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.615910    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"]
Jul 29 00:19:38.616149 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.616124    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7" podUID="5a4cabbe-c867-48fe-93da-84e081f5eb13" containerName="payload-pre-processing" containerID="cri-o://e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b" gracePeriod=30
Jul 29 00:19:38.887258 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.887231    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:19:38.890026 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.890004    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:19:38.993578 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.993545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4cabbe-c867-48fe-93da-84e081f5eb13-plugins-config-volume\") pod \"5a4cabbe-c867-48fe-93da-84e081f5eb13\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") "
Jul 29 00:19:38.993766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.993584    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flwrc\" (UniqueName: \"kubernetes.io/projected/5f6a28c7-0130-4df3-99ef-0679a09f9399-kube-api-access-flwrc\") pod \"5f6a28c7-0130-4df3-99ef-0679a09f9399\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") "
Jul 29 00:19:38.993766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.993676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6a28c7-0130-4df3-99ef-0679a09f9399-plugins-config-volume\") pod \"5f6a28c7-0130-4df3-99ef-0679a09f9399\" (UID: \"5f6a28c7-0130-4df3-99ef-0679a09f9399\") "
Jul 29 00:19:38.993766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.993701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgprr\" (UniqueName: \"kubernetes.io/projected/5a4cabbe-c867-48fe-93da-84e081f5eb13-kube-api-access-cgprr\") pod \"5a4cabbe-c867-48fe-93da-84e081f5eb13\" (UID: \"5a4cabbe-c867-48fe-93da-84e081f5eb13\") "
Jul 29 00:19:38.994009 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.993982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4cabbe-c867-48fe-93da-84e081f5eb13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a4cabbe-c867-48fe-93da-84e081f5eb13" (UID: "5a4cabbe-c867-48fe-93da-84e081f5eb13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:19:38.994067 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.994054    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6a28c7-0130-4df3-99ef-0679a09f9399-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f6a28c7-0130-4df3-99ef-0679a09f9399" (UID: "5f6a28c7-0130-4df3-99ef-0679a09f9399"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 00:19:38.995718 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.995689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6a28c7-0130-4df3-99ef-0679a09f9399-kube-api-access-flwrc" (OuterVolumeSpecName: "kube-api-access-flwrc") pod "5f6a28c7-0130-4df3-99ef-0679a09f9399" (UID: "5f6a28c7-0130-4df3-99ef-0679a09f9399"). InnerVolumeSpecName "kube-api-access-flwrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:19:38.995817 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:38.995749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4cabbe-c867-48fe-93da-84e081f5eb13-kube-api-access-cgprr" (OuterVolumeSpecName: "kube-api-access-cgprr") pod "5a4cabbe-c867-48fe-93da-84e081f5eb13" (UID: "5a4cabbe-c867-48fe-93da-84e081f5eb13"). InnerVolumeSpecName "kube-api-access-cgprr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 00:19:39.094510 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.094469    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4cabbe-c867-48fe-93da-84e081f5eb13-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:39.094510 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.094500    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flwrc\" (UniqueName: \"kubernetes.io/projected/5f6a28c7-0130-4df3-99ef-0679a09f9399-kube-api-access-flwrc\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:39.094510 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.094510    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6a28c7-0130-4df3-99ef-0679a09f9399-plugins-config-volume\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:39.094510 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.094519    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgprr\" (UniqueName: \"kubernetes.io/projected/5a4cabbe-c867-48fe-93da-84e081f5eb13-kube-api-access-cgprr\") on node \"ip-10-0-135-158.ec2.internal\" DevicePath \"\""
Jul 29 00:19:39.293831 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.293734    2581 generic.go:356] "Generic (PLEG): container finished" podID="5f6a28c7-0130-4df3-99ef-0679a09f9399" containerID="adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343" exitCode=0
Jul 29 00:19:39.293831 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.293771    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"
Jul 29 00:19:39.293831 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.293818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs" event={"ID":"5f6a28c7-0130-4df3-99ef-0679a09f9399","Type":"ContainerDied","Data":"adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343"}
Jul 29 00:19:39.294137 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.293879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs" event={"ID":"5f6a28c7-0130-4df3-99ef-0679a09f9399","Type":"ContainerDied","Data":"938706a3e7ef2b915d90d1c4041b8137c365cca61373494ec878ae5a4647b4db"}
Jul 29 00:19:39.294137 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.293909    2581 scope.go:117] "RemoveContainer" containerID="adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343"
Jul 29 00:19:39.295185 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.295166    2581 generic.go:356] "Generic (PLEG): container finished" podID="5a4cabbe-c867-48fe-93da-84e081f5eb13" containerID="e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b" exitCode=0
Jul 29 00:19:39.295269 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.295195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7" event={"ID":"5a4cabbe-c867-48fe-93da-84e081f5eb13","Type":"ContainerDied","Data":"e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b"}
Jul 29 00:19:39.295269 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.295207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"
Jul 29 00:19:39.295269 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.295231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7" event={"ID":"5a4cabbe-c867-48fe-93da-84e081f5eb13","Type":"ContainerDied","Data":"d1a17c5eb5280378edb68319611d58d3bc5964713fe66e99dce4625dba07e0be"}
Jul 29 00:19:39.303028 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.302899    2581 scope.go:117] "RemoveContainer" containerID="adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343"
Jul 29 00:19:39.303356 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:19:39.303321    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343\": container with ID starting with adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343 not found: ID does not exist" containerID="adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343"
Jul 29 00:19:39.303454 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.303361    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343"} err="failed to get container status \"adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343\": rpc error: code = NotFound desc = could not find container \"adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343\": container with ID starting with adde4ed57beb0775babe8fc95014bfceac4f8ed061545759d80704c67ca43343 not found: ID does not exist"
Jul 29 00:19:39.303454 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.303386    2581 scope.go:117] "RemoveContainer" containerID="e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b"
Jul 29 00:19:39.311465 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.311449    2581 scope.go:117] "RemoveContainer" containerID="e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b"
Jul 29 00:19:39.311772 ip-10-0-135-158 kubenswrapper[2581]: E0729 00:19:39.311754    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b\": container with ID starting with e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b not found: ID does not exist" containerID="e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b"
Jul 29 00:19:39.311832 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.311779    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b"} err="failed to get container status \"e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b\": rpc error: code = NotFound desc = could not find container \"e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b\": container with ID starting with e02ae99777c8c389b5f6531c621756d31ad2916362e24cbba01cbdf5ff5cec0b not found: ID does not exist"
Jul 29 00:19:39.318285 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.318263    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"]
Jul 29 00:19:39.322449 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.322428    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-fa6130-6486659b9b-7qtr7"]
Jul 29 00:19:39.334857 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.334830    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"]
Jul 29 00:19:39.337230 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:39.337206    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-fa6130-6659cd7dd6-nkvbs"]
Jul 29 00:19:40.981727 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:40.981693    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4cabbe-c867-48fe-93da-84e081f5eb13" path="/var/lib/kubelet/pods/5a4cabbe-c867-48fe-93da-84e081f5eb13/volumes"
Jul 29 00:19:40.982107 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:40.982056    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f6a28c7-0130-4df3-99ef-0679a09f9399" path="/var/lib/kubelet/pods/5f6a28c7-0130-4df3-99ef-0679a09f9399/volumes"
Jul 29 00:19:41.717970 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:41.717932    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ctkwd"]
Jul 29 00:19:51.880168 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:51.880140    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dc9c5fb48-7hw5m_dcd05d2a-9932-4c89-b1d3-68c0f4f0115d/copy-manifests/0.log"
Jul 29 00:19:51.886795 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:51.886771    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dc9c5fb48-7hw5m_dcd05d2a-9932-4c89-b1d3-68c0f4f0115d/manager/0.log"
Jul 29 00:19:54.387694 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:54.387660    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zlvzq_ace06479-a5a9-480e-b560-c6875c2039ed/manager/0.log"
Jul 29 00:19:54.494592 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:54.494562    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dbgc5_63112f2b-fcdd-41e2-9de1-151def88d354/kuadrant-console-plugin/0.log"
Jul 29 00:19:54.841235 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:54.841160    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ctkwd_80689370-2339-4850-aa95-eac9cff6c354/limitador/0.log"
Jul 29 00:19:55.520189 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:55.520162    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f8c77c69-896nb_4c8fb409-49d6-453f-9c3e-e34db8aa6eb2/kube-auth-proxy/0.log"
Jul 29 00:19:56.675116 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:56.675085    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9dmmv_e989e357-dbee-4c4b-9a35-2c2346d272a2/main/0.log"
Jul 29 00:19:56.996889 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:19:56.996805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xjzqp_64c28241-4e7c-4edf-ba09-7e6db4fba9ba/main/0.log"
Jul 29 00:20:05.353833 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:05.353805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fkq46_17ed82b0-64ac-4fe9-aff4-3ef02f1476a0/global-pull-secret-syncer/0.log"
Jul 29 00:20:05.469594 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:05.469566    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-86l88_d6fb9b58-ad38-4117-88c6-8465b0ba3496/konnectivity-agent/0.log"
Jul 29 00:20:05.536546 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:05.536515    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-158.ec2.internal_a0594a384a49868ef7a8441f356bc9f2/haproxy/0.log"
Jul 29 00:20:10.846266 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:10.846230    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zlvzq_ace06479-a5a9-480e-b560-c6875c2039ed/manager/0.log"
Jul 29 00:20:10.873641 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:10.873603    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dbgc5_63112f2b-fcdd-41e2-9de1-151def88d354/kuadrant-console-plugin/0.log"
Jul 29 00:20:11.178638 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:11.178593    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ctkwd_80689370-2339-4850-aa95-eac9cff6c354/limitador/0.log"
Jul 29 00:20:13.200366 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.200340    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-tlmkg_0d35dd16-c51a-485c-acd1-88eaac50ac78/cluster-monitoring-operator/0.log"
Jul 29 00:20:13.331168 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.331076    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-t9qrj_8ec62df0-dfa1-453c-8987-4d1c7fde37d2/monitoring-plugin/0.log"
Jul 29 00:20:13.538250 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.538221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zcx2f_17883e43-b87e-4bbd-8756-3455beed4353/node-exporter/0.log"
Jul 29 00:20:13.557519 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.557493    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zcx2f_17883e43-b87e-4bbd-8756-3455beed4353/kube-rbac-proxy/0.log"
Jul 29 00:20:13.580789 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.580764    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zcx2f_17883e43-b87e-4bbd-8756-3455beed4353/init-textfile/0.log"
Jul 29 00:20:13.692405 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.692375    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6540b711-08f7-48cc-9f7a-668d98d894e6/prometheus/0.log"
Jul 29 00:20:13.712600 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.712577    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6540b711-08f7-48cc-9f7a-668d98d894e6/config-reloader/0.log"
Jul 29 00:20:13.735083 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.735064    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6540b711-08f7-48cc-9f7a-668d98d894e6/thanos-sidecar/0.log"
Jul 29 00:20:13.754954 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.754934    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6540b711-08f7-48cc-9f7a-668d98d894e6/kube-rbac-proxy-web/0.log"
Jul 29 00:20:13.776423 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.776402    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6540b711-08f7-48cc-9f7a-668d98d894e6/kube-rbac-proxy/0.log"
Jul 29 00:20:13.797115 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.797096    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6540b711-08f7-48cc-9f7a-668d98d894e6/kube-rbac-proxy-thanos/0.log"
Jul 29 00:20:13.817479 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.817463    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6540b711-08f7-48cc-9f7a-668d98d894e6/init-config-reloader/0.log"
Jul 29 00:20:13.950263 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.950185    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ff44949c6-hmkfn_f2161bd0-9deb-4a83-a2e3-bd2929388726/telemeter-client/0.log"
Jul 29 00:20:13.970918 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.970890    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ff44949c6-hmkfn_f2161bd0-9deb-4a83-a2e3-bd2929388726/reload/0.log"
Jul 29 00:20:13.990199 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:13.990171    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ff44949c6-hmkfn_f2161bd0-9deb-4a83-a2e3-bd2929388726/kube-rbac-proxy/0.log"
Jul 29 00:20:14.019675 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:14.019642    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576699c6dd-2lphd_debb10cc-5bef-44e1-a480-e7969d71a7d7/thanos-query/0.log"
Jul 29 00:20:14.041500 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:14.041468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576699c6dd-2lphd_debb10cc-5bef-44e1-a480-e7969d71a7d7/kube-rbac-proxy-web/0.log"
Jul 29 00:20:14.061604 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:14.061568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576699c6dd-2lphd_debb10cc-5bef-44e1-a480-e7969d71a7d7/kube-rbac-proxy/0.log"
Jul 29 00:20:14.080948 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:14.080918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576699c6dd-2lphd_debb10cc-5bef-44e1-a480-e7969d71a7d7/prom-label-proxy/0.log"
Jul 29 00:20:14.106593 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:14.106568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576699c6dd-2lphd_debb10cc-5bef-44e1-a480-e7969d71a7d7/kube-rbac-proxy-rules/0.log"
Jul 29 00:20:14.126423 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:14.126398    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-576699c6dd-2lphd_debb10cc-5bef-44e1-a480-e7969d71a7d7/kube-rbac-proxy-metrics/0.log"
Jul 29 00:20:15.306202 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.306173    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-h4vwh_5af2f9d8-3754-4601-983c-f6216aac6937/networking-console-plugin/0.log"
Jul 29 00:20:15.720668 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.720622    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"]
Jul 29 00:20:15.721064 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721047    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" containerName="payload-pre-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721069    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" containerName="payload-pre-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721081    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" containerName="payload-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721090    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" containerName="payload-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721106    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4cabbe-c867-48fe-93da-84e081f5eb13" containerName="payload-pre-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721115    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4cabbe-c867-48fe-93da-84e081f5eb13" containerName="payload-pre-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721126    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eacfab7-d6cd-4222-b2ec-3af905ffd018" containerName="payload-pre-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721136    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eacfab7-d6cd-4222-b2ec-3af905ffd018" containerName="payload-pre-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721148    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0f134f-3895-40ea-949f-b49ce8b7020d" containerName="payload-pre-processing"
Jul 29 00:20:15.721160 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721156    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0f134f-3895-40ea-949f-b49ce8b7020d" containerName="payload-pre-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721167    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6947d6b8-e041-4969-b400-6c365d2d46c3" containerName="payload-pre-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721176    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6947d6b8-e041-4969-b400-6c365d2d46c3" containerName="payload-pre-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721191    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb603b92-d906-4547-9dc0-a6a9824af79c" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721200    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb603b92-d906-4547-9dc0-a6a9824af79c" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721214    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c24a3b7-9e63-4133-97c9-242513b40d17" containerName="main"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721224    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c24a3b7-9e63-4133-97c9-242513b40d17" containerName="main"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721237    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721245    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721259    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e07381-904e-4847-b3a5-3ce4d0cbba47" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721267    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e07381-904e-4847-b3a5-3ce4d0cbba47" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721279    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f6a28c7-0130-4df3-99ef-0679a09f9399" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721289    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f6a28c7-0130-4df3-99ef-0679a09f9399" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721388    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c24a3b7-9e63-4133-97c9-242513b40d17" containerName="main"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721401    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f34dc0-5b9c-4fc5-9f15-83b74a0984a1" containerName="payload-pre-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721411    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e07381-904e-4847-b3a5-3ce4d0cbba47" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721419    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd08e21e-f7fe-44f9-8682-fa6cb395c1a1" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721431    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0f134f-3895-40ea-949f-b49ce8b7020d" containerName="payload-pre-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721441    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f6a28c7-0130-4df3-99ef-0679a09f9399" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721450    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6947d6b8-e041-4969-b400-6c365d2d46c3" containerName="payload-pre-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721460    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0ddcf9-eb72-4d24-8ce8-7acb01019a28" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721470    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb603b92-d906-4547-9dc0-a6a9824af79c" containerName="payload-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721479    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4cabbe-c867-48fe-93da-84e081f5eb13" containerName="payload-pre-processing"
Jul 29 00:20:15.721605 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.721487    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eacfab7-d6cd-4222-b2ec-3af905ffd018" containerName="payload-pre-processing"
Jul 29 00:20:15.725068 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.725047    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.727653 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.727619    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bwncz\"/\"kube-root-ca.crt\""
Jul 29 00:20:15.728562 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.728545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bwncz\"/\"openshift-service-ca.crt\""
Jul 29 00:20:15.728673 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.728571    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bwncz\"/\"default-dockercfg-42lcq\""
Jul 29 00:20:15.735093 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.735071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"]
Jul 29 00:20:15.823536 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.823505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mfh\" (UniqueName: \"kubernetes.io/projected/572a2fd7-5b7a-41ae-8fc1-338333786881-kube-api-access-v8mfh\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.823536 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.823538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-lib-modules\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.823776 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.823560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-podres\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.823776 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.823711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-proc\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.823776 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.823767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-sys\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924553 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-sys\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mfh\" (UniqueName: \"kubernetes.io/projected/572a2fd7-5b7a-41ae-8fc1-338333786881-kube-api-access-v8mfh\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-lib-modules\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-podres\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924766 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-sys\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924971 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-podres\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924971 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-proc\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924971 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-lib-modules\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.924971 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.924883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/572a2fd7-5b7a-41ae-8fc1-338333786881-proc\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:15.932765 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:15.932737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mfh\" (UniqueName: \"kubernetes.io/projected/572a2fd7-5b7a-41ae-8fc1-338333786881-kube-api-access-v8mfh\") pod \"perf-node-gather-daemonset-f5tmh\" (UID: \"572a2fd7-5b7a-41ae-8fc1-338333786881\") " pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:16.036195 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:16.036101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:16.166667 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:16.166619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"]
Jul 29 00:20:16.169992 ip-10-0-135-158 kubenswrapper[2581]: W0729 00:20:16.169968    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod572a2fd7_5b7a_41ae_8fc1_338333786881.slice/crio-4c6e556310241a398159a753a54e3e1f6960238a4f16bd2b372fea599debf6da WatchSource:0}: Error finding container 4c6e556310241a398159a753a54e3e1f6960238a4f16bd2b372fea599debf6da: Status 404 returned error can't find the container with id 4c6e556310241a398159a753a54e3e1f6960238a4f16bd2b372fea599debf6da
Jul 29 00:20:16.468897 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:16.468864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh" event={"ID":"572a2fd7-5b7a-41ae-8fc1-338333786881","Type":"ContainerStarted","Data":"f1a52699e9c49118fedb82fe32232d72a89420a500751babef076bcf6c83485e"}
Jul 29 00:20:16.469296 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:16.468920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh" event={"ID":"572a2fd7-5b7a-41ae-8fc1-338333786881","Type":"ContainerStarted","Data":"4c6e556310241a398159a753a54e3e1f6960238a4f16bd2b372fea599debf6da"}
Jul 29 00:20:16.469296 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:16.468952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:16.485742 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:16.485695    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh" podStartSLOduration=1.48568176 podStartE2EDuration="1.48568176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:20:16.484221225 +0000 UTC m=+3511.986767757" watchObservedRunningTime="2026-07-29 00:20:16.48568176 +0000 UTC m=+3511.988228289"
Jul 29 00:20:17.173396 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:17.173372    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-685sw_0899f726-95b5-4000-8379-3753c736707d/volume-data-source-validator/0.log"
Jul 29 00:20:18.184671 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:18.184644    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c9l5_e343f41c-bdac-477f-94b8-2b41ba74100d/dns/0.log"
Jul 29 00:20:18.203839 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:18.203816    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c9l5_e343f41c-bdac-477f-94b8-2b41ba74100d/kube-rbac-proxy/0.log"
Jul 29 00:20:18.318042 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:18.318013    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w5rtn_9125b4bf-4b57-4adb-a1c6-9eb94d4525db/dns-node-resolver/0.log"
Jul 29 00:20:18.900052 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:18.900018    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d8b678dc9-6rcqc_19f3ce17-f120-42c3-9a4f-bf4aa94c7cbf/registry/0.log"
Jul 29 00:20:18.967169 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:18.967139    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvngb_0ae77f6d-9b0e-4784-8f9d-1f89fbdfdd91/node-ca/0.log"
Jul 29 00:20:20.108598 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:20.108568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f8c77c69-896nb_4c8fb409-49d6-453f-9c3e-e34db8aa6eb2/kube-auth-proxy/0.log"
Jul 29 00:20:21.110476 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:21.110448    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kbzfw_4daeb74a-6c58-4c2b-836f-73d705ad0e5b/serve-healthcheck-canary/0.log"
Jul 29 00:20:21.696785 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:21.696759    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jq58v_39d5eecd-36b6-4c80-94bf-47cb2888e4f8/kube-rbac-proxy/0.log"
Jul 29 00:20:21.715466 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:21.715439    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jq58v_39d5eecd-36b6-4c80-94bf-47cb2888e4f8/exporter/0.log"
Jul 29 00:20:21.735855 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:21.735835    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jq58v_39d5eecd-36b6-4c80-94bf-47cb2888e4f8/extractor/0.log"
Jul 29 00:20:22.482634 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:22.482600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bwncz/perf-node-gather-daemonset-f5tmh"
Jul 29 00:20:24.478112 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:24.478082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dc9c5fb48-7hw5m_dcd05d2a-9932-4c89-b1d3-68c0f4f0115d/manager/0.log"
Jul 29 00:20:24.497082 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:24.497054    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dc9c5fb48-7hw5m_dcd05d2a-9932-4c89-b1d3-68c0f4f0115d/copy-manifests/0.log"
Jul 29 00:20:26.084827 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:26.084796    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7858699d7d-qjxws_645e35bd-5b03-4ccb-b252-71d9b6e01cee/manager/0.log"
Jul 29 00:20:26.530057 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:26.530028    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f8c77c69-896nb_4c8fb409-49d6-453f-9c3e-e34db8aa6eb2/kube-auth-proxy/0.log"
Jul 29 00:20:28.214053 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:28.214024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f8c77c69-896nb_4c8fb409-49d6-453f-9c3e-e34db8aa6eb2/kube-auth-proxy/0.log"
Jul 29 00:20:31.502472 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:31.502434    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm_71381373-e286-4cf0-8f0c-5048409119e9/kube-storage-version-migrator-operator/1.log"
Jul 29 00:20:31.503490 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:31.503468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-vdvwm_71381373-e286-4cf0-8f0c-5048409119e9/kube-storage-version-migrator-operator/0.log"
Jul 29 00:20:32.664985 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:32.664954    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xxts_aae37dc3-45bf-4c81-a4af-40683d4968ec/kube-multus-additional-cni-plugins/0.log"
Jul 29 00:20:32.684955 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:32.684929    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xxts_aae37dc3-45bf-4c81-a4af-40683d4968ec/egress-router-binary-copy/0.log"
Jul 29 00:20:32.703853 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:32.703825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xxts_aae37dc3-45bf-4c81-a4af-40683d4968ec/cni-plugins/0.log"
Jul 29 00:20:32.723830 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:32.723806    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xxts_aae37dc3-45bf-4c81-a4af-40683d4968ec/bond-cni-plugin/0.log"
Jul 29 00:20:32.743425 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:32.743403    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xxts_aae37dc3-45bf-4c81-a4af-40683d4968ec/routeoverride-cni/0.log"
Jul 29 00:20:32.765319 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:32.765290    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xxts_aae37dc3-45bf-4c81-a4af-40683d4968ec/whereabouts-cni-bincopy/0.log"
Jul 29 00:20:32.788258 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:32.788235    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8xxts_aae37dc3-45bf-4c81-a4af-40683d4968ec/whereabouts-cni/0.log"
Jul 29 00:20:33.044121 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:33.044043    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h9td4_42cf06e5-5e66-4b39-9671-d88da0a957db/kube-multus/0.log"
Jul 29 00:20:33.216447 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:33.216420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m968q_0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6/network-metrics-daemon/0.log"
Jul 29 00:20:33.235823 ip-10-0-135-158 kubenswrapper[2581]: I0729 00:20:33.235799    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m968q_0b6a6c5f-b4da-42c5-9fd6-338c0a275ef6/kube-rbac-proxy/0.log"