Jul 24 17:38:11.000930 ip-10-0-142-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 17:38:11.000944 ip-10-0-142-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 17:38:11.000955 ip-10-0-142-87 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 17:38:11.001294 ip-10-0-142-87 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 17:38:20.702553 ip-10-0-142-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 24 17:38:20.702569 ip-10-0-142-87 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b68a86f840eb4f20a9eae99776681429 --
Jul 24 17:40:08.793676 ip-10-0-142-87 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 17:40:09.082684 ip-10-0-142-87 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:40:09.082684 ip-10-0-142-87 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 17:40:09.082684 ip-10-0-142-87 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:40:09.082684 ip-10-0-142-87 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 17:40:09.082684 ip-10-0-142-87 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 17:40:09.084509 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.084389    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 17:40:09.087685 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087669    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:40:09.087685 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087685    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087689    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087692    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087695    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087698    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087701    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087703    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087706    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087708    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087711    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087715    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087718    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087721    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087723    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087726    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087731    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087735    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087738    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:40:09.087755 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087740    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087748    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087751    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087755    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087758    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087761    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087763    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087766    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087769    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087772    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087774    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087777    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087779    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087781    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087784    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087787    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087789    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087792    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087794    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:40:09.088179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087799    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087802    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087806    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087808    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087811    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087814    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087816    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087819    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087821    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087823    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087826    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087829    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087831    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087834    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087837    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087841    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087845    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087849    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087852    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087854    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:40:09.088668 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087857    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087860    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087862    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087865    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087867    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087870    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087873    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087876    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087878    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087880    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087883    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087886    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087889    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087892    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087894    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087897    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087899    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087901    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087904    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087907    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:40:09.089139 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087909    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087912    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087914    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087917    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087919    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087921    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087924    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087927    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087930    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087932    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087935    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087938    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087942    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087944    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087947    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087950    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087952    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087955    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087958    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087960    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:40:09.089676 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.087963    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088689    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088696    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088699    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088702    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088705    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088708    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088711    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088714    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088717    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088720    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088722    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088725    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088728    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088730    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088733    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088735    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088738    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088741    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088744    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:40:09.090182 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088747    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088750    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088753    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088756    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088758    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088761    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088764    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088767    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088771    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088774    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088777    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088779    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088782    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088785    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088787    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088790    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088793    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088796    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088798    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:40:09.090675 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088801    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088804    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088806    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088809    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088811    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088814    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088816    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088819    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088821    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088824    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088826    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088829    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088832    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088835    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088838    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088840    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088843    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088845    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088848    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088851    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:40:09.091135 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088854    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088856    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088859    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088861    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088866    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088869    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088872    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088875    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088878    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088881    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088884    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088887    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088890    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088893    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088895    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088899    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088902    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088905    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088908    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:40:09.091656 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088911    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088914    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088916    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088920    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088923    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088926    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088928    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088931    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088933    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088936    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088939    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088942    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088944    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088947    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088949    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088952    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088954    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088957    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088959    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088962    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:40:09.092115 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088965    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.088967    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089403    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089409    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089412    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089415    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089418    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089421    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089425    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089428    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089431    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089434    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089437    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089439    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089443    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089445    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089448    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089452    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089454    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:40:09.092615 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089457    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089460    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089463    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089465    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089468    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089472    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089476    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089479    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089481    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089484    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089487    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089490    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089493    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089496    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089499    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089502    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089505    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089508    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089510    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:40:09.093068 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089513    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089516    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089519    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089521    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089524    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089526    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089529    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089531    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089533    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089536    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089539    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089542    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089545    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089547    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089551    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089553    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089555    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089558    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089561    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089563    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:40:09.093531 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089566    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089568    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089571    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089574    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089577    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089580    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089582    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089584    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089588    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089590    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089593    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089595    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089598    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089600    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089603    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089608    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089611    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089613    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089616    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089618    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:40:09.094011 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089621    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089623    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089626    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089630    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089632    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089635    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089637    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089640    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089642    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089645    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089648    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089650    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089653    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089655    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089658    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089661    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089663    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089666    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089668    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089671    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:40:09.094653 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089673    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089676    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.089678    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089750    2551 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089757    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089764    2551 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089769    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089775    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089779    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089784    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089791    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089795    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089800    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089805    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089810    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089816    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089819    2551 flags.go:64] FLAG: --cgroup-root=""
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089822    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089825    2551 flags.go:64] FLAG: --client-ca-file=""
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089827    2551 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089830    2551 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089835    2551 flags.go:64] FLAG: --cluster-domain=""
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089837    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089841    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 17:40:09.095145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089844    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089847    2551 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089851    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089854    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089857    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089860    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089863    2551 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089866    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089869    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089872    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089875    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089879    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089882    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089884    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089887    2551 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089891    2551 flags.go:64] FLAG: --enable-server="true"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089895    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089900    2551 flags.go:64] FLAG: --event-burst="100"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089903    2551 flags.go:64] FLAG: --event-qps="50"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089906    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089909    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089912    2551 flags.go:64] FLAG: --eviction-hard=""
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089915    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089920    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089923    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 17:40:09.095761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089926    2551 flags.go:64] FLAG: --eviction-soft=""
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089929    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089931    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089934    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089937    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089940    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089943    2551 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089945    2551 flags.go:64] FLAG: --feature-gates=""
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089949    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089952    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089956    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089958    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089961    2551 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089964    2551 flags.go:64] FLAG: --help="false"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089967    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089970    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089973    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089976    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089979    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089982    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089985    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089987    2551 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089992    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 17:40:09.096350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.089995    2551 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090006    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090010    2551 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090012    2551 flags.go:64] FLAG: --kube-reserved=""
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090015    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090018    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090021    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090024    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090027    2551 flags.go:64] FLAG: --lock-file=""
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090030    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090033    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090036    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090041    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090044    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090047    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090049    2551 flags.go:64] FLAG: --logging-format="text"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090052    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090055    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090058    2551 flags.go:64] FLAG: --manifest-url=""
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090061    2551 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090065    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090068    2551 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090072    2551 flags.go:64] FLAG: --max-pods="110"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090074    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090077    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 17:40:09.096942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090080    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090083    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090086    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090089    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090092    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090098    2551 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090102    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090105    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090108    2551 flags.go:64] FLAG: --pod-cidr=""
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090116    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090122    2551 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090125    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090128    2551 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090130    2551 flags.go:64] FLAG: --port="10250"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090133    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090137    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07984692cca66e7ab"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090141    2551 flags.go:64] FLAG: --qos-reserved=""
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090144    2551 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090147    2551 flags.go:64] FLAG: --register-node="true"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090149    2551 flags.go:64] FLAG: --register-with-taints=""
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090153    2551 flags.go:64] FLAG: --registry-burst="10"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090156    2551 flags.go:64] FLAG: --registry-qps="5"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090159    2551 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090161    2551 flags.go:64] FLAG: --reserved-memory=""
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090165    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 17:40:09.097608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090183    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090187    2551 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090191    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090194    2551 flags.go:64] FLAG: --runonce="false"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090197    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090200    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090203    2551 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090206    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090209    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090212    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090215    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090218    2551 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090220    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090224    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090227    2551 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090230    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090233    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090241    2551 flags.go:64] FLAG: --system-cgroups=""
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090244    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090250    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090253    2551 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090256    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090261    2551 flags.go:64] FLAG: --tls-min-version=""
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090264    2551 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090267    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 17:40:09.098254 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090269    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090272    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090275    2551 flags.go:64] FLAG: --v="2"
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090279    2551 flags.go:64] FLAG: --version="false"
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090283    2551 flags.go:64] FLAG: --vmodule=""
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090287    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.090290    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090436    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090440    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090443    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090446    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090449    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090451    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090454    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090456    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090459    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090462    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090464    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090467    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090469    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090473    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:40:09.098872 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090475    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090477    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090480    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090482    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090487    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090489    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090492    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090494    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090499    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090501    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090504    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090506    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090509    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090511    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090514    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090516    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090519    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090521    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090523    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090526    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:40:09.099394 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090528    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090531    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090533    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090536    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090538    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090541    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090543    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090546    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090549    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090551    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090554    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090556    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090559    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090562    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090564    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090567    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090571    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090575    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090578    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090580    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:40:09.099878 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090584    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090587    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090590    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090592    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090595    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090597    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090600    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090602    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090605    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090607    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090610    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090613    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090615    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090617    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090621    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090624    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090627    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090630    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090632    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:40:09.100356 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090635    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090637    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090640    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090657    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090662    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090669    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090674    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090678    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090681    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090684    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090687    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090690    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090693    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090698    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090700    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090703    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090706    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090709    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090711    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:40:09.100833 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090714    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090716    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090719    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090722    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090724    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090727    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.090729    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.091235    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.098888    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.098905    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.099494    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.099502    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099565    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099570    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099574    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:40:09.101295 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099577    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099580    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099583    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099586    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099588    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099591    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099593    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099596    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099599    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099603    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099607    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099610    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099613    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099616    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099619    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099622    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099624    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099627    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099631    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099634    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:40:09.101733 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099637    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099640    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099643    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099646    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099649    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099652    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099656    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099659    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099662    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099665    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099667    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099670    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099673    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099675    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099678    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099680    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099683    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099685    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099688    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099690    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:40:09.102210 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099693    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099695    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099698    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099700    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099703    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099706    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099708    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099711    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099713    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099716    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099718    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099721    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099724    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099726    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099730    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099733    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099735    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099738    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099741    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:40:09.102747 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099745    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099747    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099750    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099752    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099755    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099757    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099760    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099762    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099766    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099770    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099773    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099777    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099779    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099782    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099785    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099787    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099790    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099792    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099796    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:40:09.103225 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099798    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099801    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099803    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099805    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099808    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099810    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099813    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099815    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099818    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099821    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099823    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099826    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099829    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099831    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099834    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099837    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099840    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:40:09.103712 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.099842    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.099847    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100003    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100009    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100012    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100015    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100018    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100020    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100023    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100026    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100029    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100031    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100034    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100037    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100040    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 17:40:09.104126 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100042    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100045    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100047    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100050    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100052    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100055    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100058    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100062    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100065    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100068    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100071    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100074    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100077    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100080    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100082    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100086    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100088    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100091    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100094    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100096    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 17:40:09.104514 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100099    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100101    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100105    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100109    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100114    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100117    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100119    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100122    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100125    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100128    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100130    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100133    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100135    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100138    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100140    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100143    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100145    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100148    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100151    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 17:40:09.104990 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100154    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100156    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100159    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100161    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100164    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100167    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100169    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100172    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100175    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100178    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100180    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100183    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100185    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100188    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100190    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100192    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100195    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100197    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100200    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 17:40:09.105624 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100202    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100204    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100207    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100210    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100212    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100214    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100217    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100220    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100222    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100224    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100227    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100229    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100232    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100234    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100237    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100240    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100242    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100245    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100247    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100250    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 17:40:09.106082 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100253    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100255    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100258    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100261    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100263    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100266    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100268    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:09.100271    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.100275    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.100422    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.102304    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.103405    2551 server.go:1019] "Starting client certificate rotation"
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.103504    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:40:09.106578 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.103992    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 17:40:09.106904 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.105612    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:40:09.110277 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.110258    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 17:40:09.119536 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.119512    2551 log.go:25] "Validated CRI v1 runtime API"
Jul 24 17:40:09.121456 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.121440    2551 log.go:25] "Validated CRI v1 image API"
Jul 24 17:40:09.122350 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.122334    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 17:40:09.125932 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.125910    2551 fs.go:135] Filesystem UUIDs: map[5337f6b9-35bc-4f69-8cf7-370239421cf4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c6d4910c-ccfd-43b7-9660-71a9df458e7f:/dev/nvme0n1p3]
Jul 24 17:40:09.126004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.125930    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 17:40:09.129245 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.129224    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:40:09.131322 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.131212    2551 manager.go:216] Machine: {Timestamp:2026-07-24 17:40:09.130078201 +0000 UTC m=+0.247005533 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3051013 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec221c3e4ce2c078c07eaf2a251dfbc5 SystemUUID:ec221c3e-4ce2-c078-c07e-af2a251dfbc5 BootID:b68a86f8-40eb-4f20-a9ea-e99776681429 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7a:20:53:0e:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7a:20:53:0e:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:7a:d2:26:01:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 17:40:09.131322 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.131319    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 17:40:09.131458 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.131429    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 17:40:09.133773 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.133748    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 17:40:09.133913 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.133775    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 17:40:09.134734 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.134724    2551 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 17:40:09.134764 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.134738    2551 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 17:40:09.134764 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.134745    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:40:09.135265 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.135255    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 17:40:09.136300 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.136291    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 17:40:09.138755 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.138744    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:40:09.138873 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.138863    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 17:40:09.138903 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.138895    2551 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 17:40:09.138930 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.138906    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 17:40:09.138930 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.138917    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 17:40:09.138930 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.138926    2551 kubelet.go:398] "Adding apiserver pod source"
Jul 24 17:40:09.139015 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.138934    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 17:40:09.139406 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.139393    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:40:09.139447 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.139421    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 17:40:09.141849 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.141832    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 17:40:09.143445 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.143431    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 17:40:09.143492 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.143462    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 17:40:09.145150 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145138    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 17:40:09.145192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145157    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 17:40:09.145192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145163    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 17:40:09.145192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145170    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 17:40:09.145192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145178    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 17:40:09.145192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145183    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 17:40:09.145192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145190    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 17:40:09.145340 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145195    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 17:40:09.145340 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145203    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 17:40:09.145340 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145209    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 17:40:09.145340 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145218    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 17:40:09.145340 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145233    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 17:40:09.145826 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145816    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 17:40:09.145826 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.145826    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 17:40:09.147444 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.147424    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 17:40:09.147549 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.147488    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 17:40:09.147549 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.147527    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 17:40:09.149026 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.149014    2551 server.go:1268] "Started kubelet"
Jul 24 17:40:09.149780 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.149365    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 17:40:09.149860 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.149724    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 17:40:09.149860 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.149833    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 17:40:09.150231 ip-10-0-142-87 systemd[1]: Started Kubernetes Kubelet.
Jul 24 17:40:09.153367 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.151066    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 17:40:09.153367 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.152541    2551 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 17:40:09.153639 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.153541    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-prz4r"
Jul 24 17:40:09.156446 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.156424    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 17:40:09.158315 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.158299    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 17:40:09.158400 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.158310    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 17:40:09.158451 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.158416    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.158768 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.158741    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 17:40:09.158842 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.158748    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 17:40:09.158842 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.158820    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 17:40:09.158991 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.158971    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-prz4r"
Jul 24 17:40:09.159129 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.158986    2551 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 17:40:09.159214 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159205    2551 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 17:40:09.159302 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159148    2551 factory.go:55] Registering systemd factory
Jul 24 17:40:09.159410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159307    2551 factory.go:223] Registration of the systemd container factory successfully
Jul 24 17:40:09.159720 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159705    2551 factory.go:153] Registering CRI-O factory
Jul 24 17:40:09.159780 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159728    2551 factory.go:223] Registration of the crio container factory successfully
Jul 24 17:40:09.159816 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159789    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 17:40:09.159816 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159811    2551 factory.go:103] Registering Raw factory
Jul 24 17:40:09.159874 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.159823    2551 manager.go:1196] Started watching for new ooms in manager
Jul 24 17:40:09.160313 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.160299    2551 manager.go:349] Starting recovery of all containers
Jul 24 17:40:09.168851 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.168829    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:40:09.170999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.170982    2551 manager.go:354] Recovery completed
Jul 24 17:40:09.174173 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.174155    2551 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-87.ec2.internal\" not found" node="ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.176036 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.176024    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:40:09.178643 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.178628    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:40:09.178716 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.178662    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:40:09.178716 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.178678    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:40:09.179257 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.179244    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 17:40:09.179257 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.179254    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:40:09.179456 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.179255    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 17:40:09.179456 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.179317    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 17:40:09.180504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.180487    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:40:09.180644 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.180625    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:40:09.180728 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.180654    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:40:09.180898 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.180885    2551 policy_none.go:49] "None policy: Start"
Jul 24 17:40:09.181001 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.180992    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 17:40:09.181079 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.181068    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 17:40:09.182565 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.182550    2551 policy_none.go:47] "Start"
Jul 24 17:40:09.215054 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.214910    2551 manager.go:337] "Starting Device Plugin manager"
Jul 24 17:40:09.215132 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.215075    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 17:40:09.215132 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.215088    2551 server.go:87] "Starting device plugin registration server"
Jul 24 17:40:09.215319 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.215309    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 17:40:09.215348 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.215322    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 17:40:09.215451 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.215431    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 17:40:09.215554 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.215518    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 17:40:09.215554 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.215527    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 17:40:09.215940 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.215924    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 17:40:09.216015 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.215976    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.292229 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.292190    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 17:40:09.293586 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.293570    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 17:40:09.293671 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.293616    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 17:40:09.293671 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.293651    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 17:40:09.293763 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.293691    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 17:40:09.296529 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.296507    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:40:09.315736 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.315715    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:40:09.317959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.317942    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:40:09.318024 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.317975    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:40:09.318024 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.317989    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:40:09.318024 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.318020    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.326167 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.326153    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.326238 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.326173    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-87.ec2.internal\": node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.339436 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.339361    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.394887 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.394852    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal"]
Jul 24 17:40:09.394982 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.394932    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:40:09.395926 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.395912    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:40:09.395998 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.395943    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:40:09.395998 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.395955    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:40:09.396987 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.396975    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:40:09.397120 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397099    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.397154 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397146    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:40:09.397696 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397679    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:40:09.397800 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397703    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:40:09.397800 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397730    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:40:09.397800 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397739    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:40:09.397800 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397709    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:40:09.397987 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.397804    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:40:09.398807 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.398793    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.398863 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.398816    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 17:40:09.399544 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.399529    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 17:40:09.399638 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.399558    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 17:40:09.399638 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.399572    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeHasSufficientPID"
Jul 24 17:40:09.424064 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.424043    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-87.ec2.internal\" not found" node="ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.428503 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.428487    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-87.ec2.internal\" not found" node="ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.440479 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.440463    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.461211 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.461191    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/64810f671791bc778efa2c22ce5dc6ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal\" (UID: \"64810f671791bc778efa2c22ce5dc6ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.461296 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.461214    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64810f671791bc778efa2c22ce5dc6ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal\" (UID: \"64810f671791bc778efa2c22ce5dc6ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.461296 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.461233    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a6f6531a3f15fe4c59b4f31695fab0d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-87.ec2.internal\" (UID: \"6a6f6531a3f15fe4c59b4f31695fab0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.541016 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.540989    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.561495 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.561472    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/64810f671791bc778efa2c22ce5dc6ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal\" (UID: \"64810f671791bc778efa2c22ce5dc6ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.561601 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.561500    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64810f671791bc778efa2c22ce5dc6ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal\" (UID: \"64810f671791bc778efa2c22ce5dc6ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.561601 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.561517    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a6f6531a3f15fe4c59b4f31695fab0d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-87.ec2.internal\" (UID: \"6a6f6531a3f15fe4c59b4f31695fab0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.561601 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.561570    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6a6f6531a3f15fe4c59b4f31695fab0d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-87.ec2.internal\" (UID: \"6a6f6531a3f15fe4c59b4f31695fab0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.561601 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.561590    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64810f671791bc778efa2c22ce5dc6ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal\" (UID: \"64810f671791bc778efa2c22ce5dc6ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.561745 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.561591    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/64810f671791bc778efa2c22ce5dc6ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal\" (UID: \"64810f671791bc778efa2c22ce5dc6ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.641907 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.641843    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.726357 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.726320    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.732010 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.731995    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:09.742332 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.742312    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.842792 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.842756    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.943285 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:09.943209    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:09.999068 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:09.999046    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:40:10.043802 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:10.043776    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:10.103142 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.103112    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 17:40:10.103602 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.103237    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 17:40:10.103602 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.103278    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 17:40:10.103602 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.103282    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 17:40:10.144571 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:10.144553    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:10.162768 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.162736    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 17:35:09 +0000 UTC" deadline="2028-01-17 09:13:21.920313153 +0000 UTC"
Jul 24 17:40:10.162768 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.162765    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12999h33m11.757551713s"
Jul 24 17:40:10.230190 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:10.230140    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64810f671791bc778efa2c22ce5dc6ce.slice/crio-47726259e7d5c8876a650e2303ccd2f8c34e7b58fad2176864f90cf794555f4c WatchSource:0}: Error finding container 47726259e7d5c8876a650e2303ccd2f8c34e7b58fad2176864f90cf794555f4c: Status 404 returned error can't find the container with id 47726259e7d5c8876a650e2303ccd2f8c34e7b58fad2176864f90cf794555f4c
Jul 24 17:40:10.230431 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:10.230408    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6f6531a3f15fe4c59b4f31695fab0d.slice/crio-b8ed3d5c52ab4bcc42f492b2621f6f061752b644640b8e555f72f63816882015 WatchSource:0}: Error finding container b8ed3d5c52ab4bcc42f492b2621f6f061752b644640b8e555f72f63816882015: Status 404 returned error can't find the container with id b8ed3d5c52ab4bcc42f492b2621f6f061752b644640b8e555f72f63816882015
Jul 24 17:40:10.235400 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.235359    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:40:10.244953 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:10.244935    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:10.297515 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.297472    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal" event={"ID":"6a6f6531a3f15fe4c59b4f31695fab0d","Type":"ContainerStarted","Data":"b8ed3d5c52ab4bcc42f492b2621f6f061752b644640b8e555f72f63816882015"}
Jul 24 17:40:10.298466 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.298438    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal" event={"ID":"64810f671791bc778efa2c22ce5dc6ce","Type":"ContainerStarted","Data":"47726259e7d5c8876a650e2303ccd2f8c34e7b58fad2176864f90cf794555f4c"}
Jul 24 17:40:10.345643 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:10.345613    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-87.ec2.internal\" not found"
Jul 24 17:40:10.349782 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.349765    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:40:10.359452 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.359436    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:10.370288 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.370267    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:40:10.371039 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.371028    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal"
Jul 24 17:40:10.378799 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:10.378787    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 17:40:11.140458 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.140424    2551 apiserver.go:52] "Watching apiserver"
Jul 24 17:40:11.148434 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.148406    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 17:40:11.148819 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.148791    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-f2hht","kube-system/global-pull-secret-syncer-f5gq5","kube-system/konnectivity-agent-q2fwc","kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59","openshift-cluster-node-tuning-operator/tuned-lc894","openshift-dns/node-resolver-84cpg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal","openshift-multus/multus-additional-cni-plugins-bnscl","openshift-image-registry/node-ca-5x8nc","openshift-multus/multus-n5qts","openshift-multus/network-metrics-daemon-wnsct","openshift-network-diagnostics/network-check-target-46rhf","openshift-network-operator/iptables-alerter-qt655"]
Jul 24 17:40:11.150135 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.150117    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.151129 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.151108    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:11.151227 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.151187    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:11.152243 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.152218    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.152901 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.152672    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 17:40:11.152901 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.152673    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 17:40:11.152901 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.152699    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 17:40:11.152901 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.152758    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rcrbd\""
Jul 24 17:40:11.153586 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.153568    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.154459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.154264    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 17:40:11.154459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.154353    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8dt9x\""
Jul 24 17:40:11.154459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.154357    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 17:40:11.154641 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.154633    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.155370 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.155339    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:40:11.155528 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.155509    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qfgz4\""
Jul 24 17:40:11.155611 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.155541    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 17:40:11.155667 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.155623    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.155932 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.155915    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 17:40:11.156825 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.156368    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 17:40:11.156825 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.156416    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 17:40:11.156825 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.156487    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 17:40:11.156825 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.156689    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ksktd\""
Jul 24 17:40:11.157228 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.157209    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.158143 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.157782    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 17:40:11.158143 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.157865    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:40:11.158143 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.158120    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d8tkk\""
Jul 24 17:40:11.159621 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.159601    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 17:40:11.160231 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.160212    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.160559 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.160538    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ttg2z\""
Jul 24 17:40:11.160644 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.160577    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 17:40:11.160780 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.160764    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 17:40:11.161069 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.161049    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 17:40:11.161185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.161103    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 17:40:11.161185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.161129    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 17:40:11.162658 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.162364    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.162658 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.162441    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 17:40:11.162658 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.162466    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:11.162800 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.162771    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-snznd\""
Jul 24 17:40:11.163307 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.163270    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 17:40:11.163727 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.163708    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.164989 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.164969    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.165842 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.165821    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 17:40:11.165965 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.165857    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wb2tl\""
Jul 24 17:40:11.166182 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.166163    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 17:40:11.166269 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.166191    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 17:40:11.166269 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.166243    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 17:40:11.166427 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.166310    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 17:40:11.166513 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.166497    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 17:40:11.166803 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.166775    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q658w\""
Jul 24 17:40:11.166892 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.166855    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:11.166950 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.166915    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:11.167252 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.167236    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 17:40:11.169458 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169438    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-var-lib-kubelet\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.169550 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169467    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/048a940e-53f5-4bf4-816f-769a02c7280b-tmp\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.169550 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169485    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpcw\" (UniqueName: \"kubernetes.io/projected/38c17424-6bce-4314-a59d-e0034753af29-kube-api-access-dxpcw\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.169550 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169511    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-iptables-alerter-script\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.169550 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169539    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.169730 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169562    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-systemd\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.169730 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169586    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:11.169730 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169616    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-kube-api-access-tqgnf\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.169730 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169642    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.169730 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169664    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzql\" (UniqueName: \"kubernetes.io/projected/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-kube-api-access-5pzql\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.169730 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169688    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38c17424-6bce-4314-a59d-e0034753af29-serviceca\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.169730 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169708    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-run\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169731    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-host\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169758    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/048a940e-53f5-4bf4-816f-769a02c7280b-etc-tuned\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169780    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169806    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-host-slash\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169828    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-registration-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169854    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-kubernetes\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b44312a-d41c-4158-addc-4fd4b7a0c61f-hosts-file\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169907    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d346c91f-4715-48f3-873f-a9cb78c34f74-kubelet-config\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169941    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d346c91f-4715-48f3-873f-a9cb78c34f74-dbus\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169966    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-socket-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.170004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.169990    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-os-release\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170021    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-modprobe-d\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170045    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysctl-conf\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170068    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-sys-fs\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170095    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7262267a-f238-42c0-944c-ba2f53479a7b-agent-certs\") pod \"konnectivity-agent-q2fwc\" (UID: \"7262267a-f238-42c0-944c-ba2f53479a7b\") " pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170119    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-device-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170146    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc54w\" (UniqueName: \"kubernetes.io/projected/0dc17ebe-0652-4923-a6f2-14934a79ac61-kube-api-access-tc54w\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170171    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-system-cni-dir\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170196    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8g8\" (UniqueName: \"kubernetes.io/projected/048a940e-53f5-4bf4-816f-769a02c7280b-kube-api-access-wm8g8\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170218    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysconfig\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170239    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysctl-d\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-sys\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170273    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-lib-modules\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170287    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b44312a-d41c-4158-addc-4fd4b7a0c61f-tmp-dir\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170309    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170325    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7262267a-f238-42c0-944c-ba2f53479a7b-konnectivity-ca\") pod \"konnectivity-agent-q2fwc\" (UID: \"7262267a-f238-42c0-944c-ba2f53479a7b\") " pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.170410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170340    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cni-binary-copy\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.171050 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.171050 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.170987    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.171050 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.171022    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38c17424-6bce-4314-a59d-e0034753af29-host\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.171050 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.171047    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zth2\" (UniqueName: \"kubernetes.io/projected/1b44312a-d41c-4158-addc-4fd4b7a0c61f-kube-api-access-6zth2\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.171225 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.171069    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cnibin\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.172433 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.172339    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 17:40:11.201279 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.201260    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gbcdp"
Jul 24 17:40:11.209567 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.209541    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gbcdp"
Jul 24 17:40:11.261081 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.261052    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 17:40:11.271255 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271228    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-run-netns\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.271255 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271260    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/048a940e-53f5-4bf4-816f-769a02c7280b-tmp\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.271478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-hostroot\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.271478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271301    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-cni-netd\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.271478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271428    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-iptables-alerter-script\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.271478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271467    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271494    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-kubelet\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271510    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-daemon-config\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271526    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-kube-api-access-tqgnf\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271543    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271566    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38c17424-6bce-4314-a59d-e0034753af29-serviceca\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271581    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-run\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271601    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/048a940e-53f5-4bf4-816f-769a02c7280b-etc-tuned\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.271635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271623    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-cni-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271639    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271649    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-kubelet\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271676    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-cni-bin\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271699    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271723    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-registration-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271746    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d346c91f-4715-48f3-873f-a9cb78c34f74-dbus\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271763    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-cnibin\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271778    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-os-release\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271793    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-systemd\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271818    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-ovn\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271855    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-os-release\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271870    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-multus-certs\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7262267a-f238-42c0-944c-ba2f53479a7b-agent-certs\") pod \"konnectivity-agent-q2fwc\" (UID: \"7262267a-f238-42c0-944c-ba2f53479a7b\") " pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271910    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-device-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271931    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc54w\" (UniqueName: \"kubernetes.io/projected/0dc17ebe-0652-4923-a6f2-14934a79ac61-kube-api-access-tc54w\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271949    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-system-cni-dir\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.271959 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271968    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-etc-kubernetes\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.271993    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkm62\" (UniqueName: \"kubernetes.io/projected/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-kube-api-access-nkm62\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272018    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlk9s\" (UniqueName: \"kubernetes.io/projected/37e312f0-8531-4f45-9267-2902b33b0788-kube-api-access-nlk9s\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-sys\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272057    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272084    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-etc-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272151    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cni-binary-copy\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272177    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272204    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272219    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-iptables-alerter-script\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272230    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-cni-multus\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-conf-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272273    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-ovnkube-config\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cnibin\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272310    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-var-lib-kubelet\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.272684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272334    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272352    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-log-socket\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272396    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-env-overrides\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272402    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272444    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpcw\" (UniqueName: \"kubernetes.io/projected/38c17424-6bce-4314-a59d-e0034753af29-kube-api-access-dxpcw\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272582    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272633    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-registration-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272721    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d346c91f-4715-48f3-873f-a9cb78c34f74-dbus\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.272818    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-os-release\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273185    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-systemd\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273232    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273263    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273326    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-netns\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273355    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzql\" (UniqueName: \"kubernetes.io/projected/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-kube-api-access-5pzql\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273399    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-host\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273424    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-socket-dir-parent\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273449    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-host-slash\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.273580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273473    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-kubernetes\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273497    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b44312a-d41c-4158-addc-4fd4b7a0c61f-hosts-file\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273525    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d346c91f-4715-48f3-873f-a9cb78c34f74-kubelet-config\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273613    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-system-cni-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273638    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-cni-binary-copy\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273662    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-ovnkube-script-lib\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273691    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-socket-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273728    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-modprobe-d\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273729    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-device-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysctl-conf\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273792    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-k8s-cni-cncf-io\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273821    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-slash\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273854    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37e312f0-8531-4f45-9267-2902b33b0788-ovn-node-metrics-cert\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273882    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-sys-fs\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273909    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8g8\" (UniqueName: \"kubernetes.io/projected/048a940e-53f5-4bf4-816f-769a02c7280b-kube-api-access-wm8g8\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.273997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274053    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-var-lib-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.274321 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274100    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-node-log\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274125    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysconfig\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysctl-d\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274169    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-lib-modules\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274233    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b44312a-d41c-4158-addc-4fd4b7a0c61f-tmp-dir\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274403    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-cni-bin\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62zz\" (UniqueName: \"kubernetes.io/projected/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-kube-api-access-z62zz\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274457    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-run-ovn-kubernetes\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274485    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7262267a-f238-42c0-944c-ba2f53479a7b-konnectivity-ca\") pod \"konnectivity-agent-q2fwc\" (UID: \"7262267a-f238-42c0-944c-ba2f53479a7b\") " pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274511    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38c17424-6bce-4314-a59d-e0034753af29-host\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274538    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zth2\" (UniqueName: \"kubernetes.io/projected/1b44312a-d41c-4158-addc-4fd4b7a0c61f-kube-api-access-6zth2\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274591    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-systemd-units\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274605    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274612    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cni-binary-copy\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274670    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-run\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274685    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cnibin\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274710    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d346c91f-4715-48f3-873f-a9cb78c34f74-kubelet-config\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.275090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274740    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-var-lib-kubelet\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274760    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-systemd\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-host\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274949    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-host-slash\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.274990    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-kubernetes\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275026    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b44312a-d41c-4158-addc-4fd4b7a0c61f-hosts-file\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275132    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-socket-dir\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275210    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-modprobe-d\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275311    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysctl-conf\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275333    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275359    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0dc17ebe-0652-4923-a6f2-14934a79ac61-sys-fs\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275405    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-system-cni-dir\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275448    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysconfig\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275487    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-etc-sysctl-d\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275582    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-lib-modules\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.275721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275677    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38c17424-6bce-4314-a59d-e0034753af29-host\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.276272 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275846    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38c17424-6bce-4314-a59d-e0034753af29-serviceca\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.276272 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275851    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b44312a-d41c-4158-addc-4fd4b7a0c61f-tmp-dir\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.276272 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275912    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/048a940e-53f5-4bf4-816f-769a02c7280b-sys\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.276272 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.275931    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7262267a-f238-42c0-944c-ba2f53479a7b-konnectivity-ca\") pod \"konnectivity-agent-q2fwc\" (UID: \"7262267a-f238-42c0-944c-ba2f53479a7b\") " pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.276272 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.275968    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:11.276272 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.276043    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret podName:d346c91f-4715-48f3-873f-a9cb78c34f74 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:11.776015841 +0000 UTC m=+2.892943163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret") pod "global-pull-secret-syncer-f5gq5" (UID: "d346c91f-4715-48f3-873f-a9cb78c34f74") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:11.276272 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.276064    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/048a940e-53f5-4bf4-816f-769a02c7280b-tmp\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.277037 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.277015    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/048a940e-53f5-4bf4-816f-769a02c7280b-etc-tuned\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.279873 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.279731    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:40:11.279873 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.279754    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:40:11.279873 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.279767    2551 projected.go:196] Error preparing data for projected volume kube-api-access-72d97 for pod openshift-network-diagnostics/network-check-target-46rhf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:11.279873 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.279816    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97 podName:86dfdc0f-5604-4754-9a47-65b0d7c910fb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:11.779799202 +0000 UTC m=+2.896726539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-72d97" (UniqueName: "kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97") pod "network-check-target-46rhf" (UID: "86dfdc0f-5604-4754-9a47-65b0d7c910fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:11.282871 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.282763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22-kube-api-access-tqgnf\") pod \"iptables-alerter-qt655\" (UID: \"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22\") " pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.282871 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.282815    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzql\" (UniqueName: \"kubernetes.io/projected/2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521-kube-api-access-5pzql\") pod \"multus-additional-cni-plugins-bnscl\" (UID: \"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521\") " pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.282871 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.282817    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc54w\" (UniqueName: \"kubernetes.io/projected/0dc17ebe-0652-4923-a6f2-14934a79ac61-kube-api-access-tc54w\") pod \"aws-ebs-csi-driver-node-gdm59\" (UID: \"0dc17ebe-0652-4923-a6f2-14934a79ac61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.283390 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.283352    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7262267a-f238-42c0-944c-ba2f53479a7b-agent-certs\") pod \"konnectivity-agent-q2fwc\" (UID: \"7262267a-f238-42c0-944c-ba2f53479a7b\") " pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.285072 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.285053    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zth2\" (UniqueName: \"kubernetes.io/projected/1b44312a-d41c-4158-addc-4fd4b7a0c61f-kube-api-access-6zth2\") pod \"node-resolver-84cpg\" (UID: \"1b44312a-d41c-4158-addc-4fd4b7a0c61f\") " pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.285216 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.285155    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpcw\" (UniqueName: \"kubernetes.io/projected/38c17424-6bce-4314-a59d-e0034753af29-kube-api-access-dxpcw\") pod \"node-ca-5x8nc\" (UID: \"38c17424-6bce-4314-a59d-e0034753af29\") " pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.286661 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.286624    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8g8\" (UniqueName: \"kubernetes.io/projected/048a940e-53f5-4bf4-816f-769a02c7280b-kube-api-access-wm8g8\") pod \"tuned-lc894\" (UID: \"048a940e-53f5-4bf4-816f-769a02c7280b\") " pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.375936 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.375907    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-cni-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.375936 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.375938    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-kubelet\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.375960    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-cni-bin\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376018    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-cni-bin\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376022    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-cni-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376019    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-kubelet\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-cnibin\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376064    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-os-release\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376083    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-cnibin\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376080    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-systemd\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376110    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-systemd\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376120    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-ovn\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376142    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-multus-certs\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376147    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-ovn\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376129    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-os-release\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376166    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-etc-kubernetes\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376186    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-etc-kubernetes\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376196    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkm62\" (UniqueName: \"kubernetes.io/projected/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-kube-api-access-nkm62\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376210    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-multus-certs\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376229    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlk9s\" (UniqueName: \"kubernetes.io/projected/37e312f0-8531-4f45-9267-2902b33b0788-kube-api-access-nlk9s\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376268    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-etc-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376300    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-cni-multus\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376367    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-conf-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376415    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-ovnkube-config\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-cni-multus\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376440    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376447    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-conf-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.376569 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376459    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376463    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-log-socket\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376498    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-log-socket\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376502    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-env-overrides\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.376529    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376423    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-etc-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.376580    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs podName:ca6bd76d-63fe-4c44-a496-6f9ab029b2cb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:11.876561573 +0000 UTC m=+2.993488912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs") pod "network-metrics-daemon-wnsct" (UID: "ca6bd76d-63fe-4c44-a496-6f9ab029b2cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376623    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376652    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-netns\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376680    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-socket-dir-parent\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376686    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-run-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-system-cni-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376735    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-cni-binary-copy\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376735    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-netns\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376754    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-socket-dir-parent\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376759    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-ovnkube-script-lib\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376786    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-system-cni-dir\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377270 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376824    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-k8s-cni-cncf-io\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376849    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-slash\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376874    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37e312f0-8531-4f45-9267-2902b33b0788-ovn-node-metrics-cert\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376899    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-var-lib-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376922    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-node-log\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376936    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-env-overrides\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376949    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-cni-bin\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62zz\" (UniqueName: \"kubernetes.io/projected/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-kube-api-access-z62zz\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376991    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-run-k8s-cni-cncf-io\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.376996    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-run-ovn-kubernetes\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377008    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-ovnkube-config\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377022    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-systemd-units\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377028    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-slash\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377053    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-systemd-units\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377059    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-run-netns\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377066    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-cni-bin\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377090    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-hostroot\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377104    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-node-log\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.377815 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377116    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-cni-netd\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-kubelet\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377157    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37e312f0-8531-4f45-9267-2902b33b0788-ovnkube-script-lib\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377172    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-daemon-config\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377197    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-hostroot\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377228    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-run-netns\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377251    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-var-lib-openvswitch\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377260    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-cni-binary-copy\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377271    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-cni-netd\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377275    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37e312f0-8531-4f45-9267-2902b33b0788-host-run-ovn-kubernetes\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377313    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-host-var-lib-kubelet\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.378687 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.377656    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-multus-daemon-config\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.379414 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.379397    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37e312f0-8531-4f45-9267-2902b33b0788-ovn-node-metrics-cert\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.387063 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.387026    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkm62\" (UniqueName: \"kubernetes.io/projected/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-kube-api-access-nkm62\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:11.387194 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.387108    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlk9s\" (UniqueName: \"kubernetes.io/projected/37e312f0-8531-4f45-9267-2902b33b0788-kube-api-access-nlk9s\") pod \"ovnkube-node-f2hht\" (UID: \"37e312f0-8531-4f45-9267-2902b33b0788\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.387194 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.387165    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62zz\" (UniqueName: \"kubernetes.io/projected/f63fe1ed-3d2a-41bb-837c-77b6beb9cb54-kube-api-access-z62zz\") pod \"multus-n5qts\" (UID: \"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54\") " pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.466434 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.466344    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5x8nc"
Jul 24 17:40:11.474944 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.474921    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:11.482056 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.482036    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:11.483091 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.483069    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:40:11.487923 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.487905    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qt655"
Jul 24 17:40:11.493819 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.493801    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59"
Jul 24 17:40:11.500224 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.500205    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lc894"
Jul 24 17:40:11.506978 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.506954    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bnscl"
Jul 24 17:40:11.513429 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.513411    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84cpg"
Jul 24 17:40:11.518184 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.518165    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5qts"
Jul 24 17:40:11.542190 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.542162    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:40:11.779557 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.779482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:11.779710 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.779657    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:11.779763 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.779720    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret podName:d346c91f-4715-48f3-873f-a9cb78c34f74 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:12.779698837 +0000 UTC m=+3.896626182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret") pod "global-pull-secret-syncer-f5gq5" (UID: "d346c91f-4715-48f3-873f-a9cb78c34f74") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:11.880633 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.880599    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:11.880633 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:11.880636    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:11.880853 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.880757    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:11.880853 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.880778    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:40:11.880853 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.880791    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:40:11.880853 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.880802    2551 projected.go:196] Error preparing data for projected volume kube-api-access-72d97 for pod openshift-network-diagnostics/network-check-target-46rhf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:11.880853 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.880824    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs podName:ca6bd76d-63fe-4c44-a496-6f9ab029b2cb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:12.880805668 +0000 UTC m=+3.997732997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs") pod "network-metrics-daemon-wnsct" (UID: "ca6bd76d-63fe-4c44-a496-6f9ab029b2cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:11.880853 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:11.880844    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97 podName:86dfdc0f-5604-4754-9a47-65b0d7c910fb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:12.880831778 +0000 UTC m=+3.997759101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-72d97" (UniqueName: "kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97") pod "network-check-target-46rhf" (UID: "86dfdc0f-5604-4754-9a47-65b0d7c910fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:12.031081 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.031015    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bbfe33_71c7_4c5a_b361_b9d2c4b6ed22.slice/crio-4c1bda4ce737950ff2505a2a0b174126fed5495b05d7e8ddba322e4348b82919 WatchSource:0}: Error finding container 4c1bda4ce737950ff2505a2a0b174126fed5495b05d7e8ddba322e4348b82919: Status 404 returned error can't find the container with id 4c1bda4ce737950ff2505a2a0b174126fed5495b05d7e8ddba322e4348b82919
Jul 24 17:40:12.036643 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.036615    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c17424_6bce_4314_a59d_e0034753af29.slice/crio-b92b879e625dc03dcc9363952d7946efab39200b2db1d1c01ce11902b2a5508c WatchSource:0}: Error finding container b92b879e625dc03dcc9363952d7946efab39200b2db1d1c01ce11902b2a5508c: Status 404 returned error can't find the container with id b92b879e625dc03dcc9363952d7946efab39200b2db1d1c01ce11902b2a5508c
Jul 24 17:40:12.038188 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.038153    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc35ddf_9a6d_41cb_8b4d_92cf7a83c521.slice/crio-75accf81c638b0b745f08c0f829b1d45a8a1681d34ba4408292f2f0785a563ae WatchSource:0}: Error finding container 75accf81c638b0b745f08c0f829b1d45a8a1681d34ba4408292f2f0785a563ae: Status 404 returned error can't find the container with id 75accf81c638b0b745f08c0f829b1d45a8a1681d34ba4408292f2f0785a563ae
Jul 24 17:40:12.040446 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.040279    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048a940e_53f5_4bf4_816f_769a02c7280b.slice/crio-35a84c9d91cca2bff6a64e92fc7fb2c78031a3296f685b632bcb44a6cc739069 WatchSource:0}: Error finding container 35a84c9d91cca2bff6a64e92fc7fb2c78031a3296f685b632bcb44a6cc739069: Status 404 returned error can't find the container with id 35a84c9d91cca2bff6a64e92fc7fb2c78031a3296f685b632bcb44a6cc739069
Jul 24 17:40:12.045980 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.045945    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc17ebe_0652_4923_a6f2_14934a79ac61.slice/crio-5f32b7ba6d9c538b193a7ea1dc88045330353a1a49476ae7b1e639f24909b358 WatchSource:0}: Error finding container 5f32b7ba6d9c538b193a7ea1dc88045330353a1a49476ae7b1e639f24909b358: Status 404 returned error can't find the container with id 5f32b7ba6d9c538b193a7ea1dc88045330353a1a49476ae7b1e639f24909b358
Jul 24 17:40:12.048843 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.048811    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63fe1ed_3d2a_41bb_837c_77b6beb9cb54.slice/crio-cbd8c0e7322dd4afe3d08534fad88839e4174191b1438bffcd47611d9093b592 WatchSource:0}: Error finding container cbd8c0e7322dd4afe3d08534fad88839e4174191b1438bffcd47611d9093b592: Status 404 returned error can't find the container with id cbd8c0e7322dd4afe3d08534fad88839e4174191b1438bffcd47611d9093b592
Jul 24 17:40:12.049511 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.049365    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e312f0_8531_4f45_9267_2902b33b0788.slice/crio-1b3e314134f49f44205cd01f1c213ef534ee95c333eb94e5ccd835eceefe0253 WatchSource:0}: Error finding container 1b3e314134f49f44205cd01f1c213ef534ee95c333eb94e5ccd835eceefe0253: Status 404 returned error can't find the container with id 1b3e314134f49f44205cd01f1c213ef534ee95c333eb94e5ccd835eceefe0253
Jul 24 17:40:12.050968 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.050941    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b44312a_d41c_4158_addc_4fd4b7a0c61f.slice/crio-b1aadfc477443570465b7ab3e2c295455e50bfe8273eacc0b7972552fe269ae6 WatchSource:0}: Error finding container b1aadfc477443570465b7ab3e2c295455e50bfe8273eacc0b7972552fe269ae6: Status 404 returned error can't find the container with id b1aadfc477443570465b7ab3e2c295455e50bfe8273eacc0b7972552fe269ae6
Jul 24 17:40:12.051976 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:40:12.051955    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7262267a_f238_42c0_944c_ba2f53479a7b.slice/crio-cdea4c4c8e77e96d2264868446d910a647699212d16e810e9aa76222c07487bd WatchSource:0}: Error finding container cdea4c4c8e77e96d2264868446d910a647699212d16e810e9aa76222c07487bd: Status 404 returned error can't find the container with id cdea4c4c8e77e96d2264868446d910a647699212d16e810e9aa76222c07487bd
Jul 24 17:40:12.211850 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.211655    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:35:11 +0000 UTC" deadline="2028-01-16 05:01:03.729321881 +0000 UTC"
Jul 24 17:40:12.211850 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.211845    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12971h20m51.517479798s"
Jul 24 17:40:12.288481 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.288392    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 17:40:12.294255 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.294236    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:12.294336 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.294260    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:12.294393 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.294328    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:12.294475 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.294457    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:12.302652 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.302626    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5qts" event={"ID":"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54","Type":"ContainerStarted","Data":"cbd8c0e7322dd4afe3d08534fad88839e4174191b1438bffcd47611d9093b592"}
Jul 24 17:40:12.303738 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.303715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lc894" event={"ID":"048a940e-53f5-4bf4-816f-769a02c7280b","Type":"ContainerStarted","Data":"35a84c9d91cca2bff6a64e92fc7fb2c78031a3296f685b632bcb44a6cc739069"}
Jul 24 17:40:12.304824 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.304807    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerStarted","Data":"75accf81c638b0b745f08c0f829b1d45a8a1681d34ba4408292f2f0785a563ae"}
Jul 24 17:40:12.305880 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.305860    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5x8nc" event={"ID":"38c17424-6bce-4314-a59d-e0034753af29","Type":"ContainerStarted","Data":"b92b879e625dc03dcc9363952d7946efab39200b2db1d1c01ce11902b2a5508c"}
Jul 24 17:40:12.308698 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.308674    2551 generic.go:356] "Generic (PLEG): container finished" podID="64810f671791bc778efa2c22ce5dc6ce" containerID="cd52ee4157f1f01d49831518f6f763bc0fcd6ea3849f492d667f810b97f9c47c" exitCode=0
Jul 24 17:40:12.308794 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.308754    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal" event={"ID":"64810f671791bc778efa2c22ce5dc6ce","Type":"ContainerDied","Data":"cd52ee4157f1f01d49831518f6f763bc0fcd6ea3849f492d667f810b97f9c47c"}
Jul 24 17:40:12.310770 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.310744    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84cpg" event={"ID":"1b44312a-d41c-4158-addc-4fd4b7a0c61f","Type":"ContainerStarted","Data":"b1aadfc477443570465b7ab3e2c295455e50bfe8273eacc0b7972552fe269ae6"}
Jul 24 17:40:12.312159 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.312137    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"1b3e314134f49f44205cd01f1c213ef534ee95c333eb94e5ccd835eceefe0253"}
Jul 24 17:40:12.313113 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.313096    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59" event={"ID":"0dc17ebe-0652-4923-a6f2-14934a79ac61","Type":"ContainerStarted","Data":"5f32b7ba6d9c538b193a7ea1dc88045330353a1a49476ae7b1e639f24909b358"}
Jul 24 17:40:12.314153 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.314135    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qt655" event={"ID":"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22","Type":"ContainerStarted","Data":"4c1bda4ce737950ff2505a2a0b174126fed5495b05d7e8ddba322e4348b82919"}
Jul 24 17:40:12.315438 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.315417    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal" event={"ID":"6a6f6531a3f15fe4c59b4f31695fab0d","Type":"ContainerStarted","Data":"bff9177d253d53d699b11d17b67718930f198ccacda0c7a8f92dcb05f7c57ccc"}
Jul 24 17:40:12.316354 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.316329    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q2fwc" event={"ID":"7262267a-f238-42c0-944c-ba2f53479a7b","Type":"ContainerStarted","Data":"cdea4c4c8e77e96d2264868446d910a647699212d16e810e9aa76222c07487bd"}
Jul 24 17:40:12.336814 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.336768    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-87.ec2.internal" podStartSLOduration=2.336754977 podStartE2EDuration="2.336754977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:12.33630756 +0000 UTC m=+3.453234901" watchObservedRunningTime="2026-07-24 17:40:12.336754977 +0000 UTC m=+3.453682317"
Jul 24 17:40:12.787920 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.787614    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:12.787920 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.787813    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:12.787920 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.787875    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret podName:d346c91f-4715-48f3-873f-a9cb78c34f74 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:14.787857791 +0000 UTC m=+5.904785126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret") pod "global-pull-secret-syncer-f5gq5" (UID: "d346c91f-4715-48f3-873f-a9cb78c34f74") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.888294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:12.888349    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.888503    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.888524    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.888537    2551 projected.go:196] Error preparing data for projected volume kube-api-access-72d97 for pod openshift-network-diagnostics/network-check-target-46rhf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.888596    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97 podName:86dfdc0f-5604-4754-9a47-65b0d7c910fb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:14.888578239 +0000 UTC m=+6.005505571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-72d97" (UniqueName: "kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97") pod "network-check-target-46rhf" (UID: "86dfdc0f-5604-4754-9a47-65b0d7c910fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.888981    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:12.889067 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:12.889029    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs podName:ca6bd76d-63fe-4c44-a496-6f9ab029b2cb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:14.889015405 +0000 UTC m=+6.005942726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs") pod "network-metrics-daemon-wnsct" (UID: "ca6bd76d-63fe-4c44-a496-6f9ab029b2cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:13.212488 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:13.212322    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:35:11 +0000 UTC" deadline="2028-02-01 18:32:39.294294399 +0000 UTC"
Jul 24 17:40:13.212488 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:13.212407    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13368h52m26.081891613s"
Jul 24 17:40:13.296932 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:13.296351    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:13.296932 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:13.296510    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:13.329210 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:13.328641    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal" event={"ID":"64810f671791bc778efa2c22ce5dc6ce","Type":"ContainerStarted","Data":"af2527dfe78f26360ab8394dd7c4c506f39661d325fe6008bed11758bbffa1a2"}
Jul 24 17:40:13.344846 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:13.343756    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-87.ec2.internal" podStartSLOduration=3.343737207 podStartE2EDuration="3.343737207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:13.343628132 +0000 UTC m=+4.460555473" watchObservedRunningTime="2026-07-24 17:40:13.343737207 +0000 UTC m=+4.460664549"
Jul 24 17:40:14.294499 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:14.294468    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:14.294956 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.294594    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:14.295014 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:14.294985    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:14.295138 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.295081    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:14.806069 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:14.806030    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:14.806278 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.806221    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:14.806348 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.806290    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret podName:d346c91f-4715-48f3-873f-a9cb78c34f74 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:18.80626984 +0000 UTC m=+9.923197161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret") pod "global-pull-secret-syncer-f5gq5" (UID: "d346c91f-4715-48f3-873f-a9cb78c34f74") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:14.907230 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:14.907149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:14.907230 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:14.907203    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:14.907542 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.907481    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:40:14.907542 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.907506    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:40:14.907542 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.907519    2551 projected.go:196] Error preparing data for projected volume kube-api-access-72d97 for pod openshift-network-diagnostics/network-check-target-46rhf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:14.907706 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.907579    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97 podName:86dfdc0f-5604-4754-9a47-65b0d7c910fb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:18.907559451 +0000 UTC m=+10.024486784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-72d97" (UniqueName: "kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97") pod "network-check-target-46rhf" (UID: "86dfdc0f-5604-4754-9a47-65b0d7c910fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:14.908041 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.908014    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:14.908137 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:14.908065    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs podName:ca6bd76d-63fe-4c44-a496-6f9ab029b2cb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:18.908049665 +0000 UTC m=+10.024977007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs") pod "network-metrics-daemon-wnsct" (UID: "ca6bd76d-63fe-4c44-a496-6f9ab029b2cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:15.296306 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:15.296227    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:15.296747 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:15.296393    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:16.294057 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:16.294019    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:16.294248 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:16.294152    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:16.294563 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:16.294543    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:16.294672 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:16.294650    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:17.294125 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:17.294092    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:17.294591 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:17.294234    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:18.295140 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:18.294437    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:18.295140 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.294570    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:18.295140 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:18.295004    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:18.295140 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.295093    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:18.843829 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:18.843790    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:18.844008 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.843992    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:18.844077 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.844054    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret podName:d346c91f-4715-48f3-873f-a9cb78c34f74 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:26.844039848 +0000 UTC m=+17.960967166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret") pod "global-pull-secret-syncer-f5gq5" (UID: "d346c91f-4715-48f3-873f-a9cb78c34f74") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:18.945500 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:18.944821    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:18.945500 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:18.944877    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:18.945500 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.945082    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:40:18.945500 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.945097    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:40:18.945500 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.945107    2551 projected.go:196] Error preparing data for projected volume kube-api-access-72d97 for pod openshift-network-diagnostics/network-check-target-46rhf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:18.945500 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.945165    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97 podName:86dfdc0f-5604-4754-9a47-65b0d7c910fb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:26.945143875 +0000 UTC m=+18.062071217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-72d97" (UniqueName: "kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97") pod "network-check-target-46rhf" (UID: "86dfdc0f-5604-4754-9a47-65b0d7c910fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:18.945500 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.945503    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:18.945976 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:18.945567    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs podName:ca6bd76d-63fe-4c44-a496-6f9ab029b2cb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:26.945551394 +0000 UTC m=+18.062478726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs") pod "network-metrics-daemon-wnsct" (UID: "ca6bd76d-63fe-4c44-a496-6f9ab029b2cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:19.295223 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:19.295144    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:19.295600 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:19.295288    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:20.294732 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:20.294702    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:20.294732 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:20.294717    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:20.294931 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:20.294816    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:20.294980 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:20.294942    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:21.294985 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:21.294946    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:21.295414 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:21.295104    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:22.294430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:22.294393    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:22.294430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:22.294422    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:22.294680 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:22.294534    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:22.294721 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:22.294676    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:23.294314 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:23.294279    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:23.294788 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:23.294419    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:24.294168 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:24.294138    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:24.294168 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:24.294149    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:24.294399 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:24.294257    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:24.294740 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:24.294403    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:25.294093 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:25.294045    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:25.294266 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:25.294217    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:26.294671 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:26.294635    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:26.295114 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:26.294653    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:26.295114 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:26.294762    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:26.295114 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:26.294866    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:26.905411 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:26.905353    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:26.905636 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:26.905523    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:26.905636 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:26.905602    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret podName:d346c91f-4715-48f3-873f-a9cb78c34f74 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:42.905580944 +0000 UTC m=+34.022508275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret") pod "global-pull-secret-syncer-f5gq5" (UID: "d346c91f-4715-48f3-873f-a9cb78c34f74") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:27.006567 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:27.006490    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:27.006567 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:27.006547    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:27.006795 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:27.006641    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:27.006795 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:27.006688    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:40:27.006795 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:27.006709    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:40:27.006795 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:27.006716    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs podName:ca6bd76d-63fe-4c44-a496-6f9ab029b2cb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:43.006697969 +0000 UTC m=+34.123625287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs") pod "network-metrics-daemon-wnsct" (UID: "ca6bd76d-63fe-4c44-a496-6f9ab029b2cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:27.006795 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:27.006723    2551 projected.go:196] Error preparing data for projected volume kube-api-access-72d97 for pod openshift-network-diagnostics/network-check-target-46rhf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:27.006795 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:27.006766    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97 podName:86dfdc0f-5604-4754-9a47-65b0d7c910fb nodeName:}" failed. No retries permitted until 2026-07-24 17:40:43.006754657 +0000 UTC m=+34.123681977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-72d97" (UniqueName: "kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97") pod "network-check-target-46rhf" (UID: "86dfdc0f-5604-4754-9a47-65b0d7c910fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:27.294192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:27.294111    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:27.294353 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:27.294250    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:28.294227 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:28.294187    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:28.294714 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:28.294305    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:28.294714 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:28.294191    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:28.294714 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:28.294695    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:29.295588 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:29.295551    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:29.296039 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:29.295691    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:30.294621 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:30.294558    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:30.294816 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:30.294566    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:30.294816 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:30.294680    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:30.294816 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:30.294785    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:31.294181 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.294005    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:31.294569 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:31.294290    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:31.369090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.369054    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59" event={"ID":"0dc17ebe-0652-4923-a6f2-14934a79ac61","Type":"ContainerStarted","Data":"e5989626b84208ce246e5bd9cb4b73436ec1405e1b75e1cd176f8e69cd88fa49"}
Jul 24 17:40:31.370485 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.370456    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q2fwc" event={"ID":"7262267a-f238-42c0-944c-ba2f53479a7b","Type":"ContainerStarted","Data":"2526d4b7361cc1f99e596a3385f12e2342411f44f21b972a03eacb110eaeb292"}
Jul 24 17:40:31.371861 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.371840    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5qts" event={"ID":"f63fe1ed-3d2a-41bb-837c-77b6beb9cb54","Type":"ContainerStarted","Data":"48764baf826cffbc33ee68d4707016e3abcda394c4bf9613db1e0227b2fb5240"}
Jul 24 17:40:31.373322 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.373304    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lc894" event={"ID":"048a940e-53f5-4bf4-816f-769a02c7280b","Type":"ContainerStarted","Data":"15486b79f190ea4a7795eeb03071ccf1af26e4edd637af3f98b1a736b2a338c6"}
Jul 24 17:40:31.377795 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.377770    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521" containerID="be1f4341468117eab5cf1893f74ac35f4a13958973d09f26d4f6d859228363a9" exitCode=0
Jul 24 17:40:31.377870 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.377839    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerDied","Data":"be1f4341468117eab5cf1893f74ac35f4a13958973d09f26d4f6d859228363a9"}
Jul 24 17:40:31.379301 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.379283    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5x8nc" event={"ID":"38c17424-6bce-4314-a59d-e0034753af29","Type":"ContainerStarted","Data":"f2ecc4386679925641bf8e23bcaf64a84de7e19cba2981cdc6038ac3e9e45035"}
Jul 24 17:40:31.380615 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.380597    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84cpg" event={"ID":"1b44312a-d41c-4158-addc-4fd4b7a0c61f","Type":"ContainerStarted","Data":"c680deba79b2e3b59b2fcc4a981634ed1b985a79a0c77ad47e8579212abc0143"}
Jul 24 17:40:31.389438 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.389395    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q2fwc" podStartSLOduration=3.767656721 podStartE2EDuration="22.389365241s" totalImagesPullingTime="18.62170852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:31.389090598 +0000 UTC m=+22.506017942" watchObservedRunningTime="2026-07-24 17:40:31.389365241 +0000 UTC m=+22.506292624"
Jul 24 17:40:31.409359 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.409313    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-84cpg" podStartSLOduration=3.7349389779999997 podStartE2EDuration="22.409292046s" totalImagesPullingTime="18.674353068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:31.408285219 +0000 UTC m=+22.525212559" watchObservedRunningTime="2026-07-24 17:40:31.409292046 +0000 UTC m=+22.526219386"
Jul 24 17:40:31.423124 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.423082    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5x8nc" podStartSLOduration=3.71303372 podStartE2EDuration="22.423066476s" totalImagesPullingTime="18.710032756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:31.422918178 +0000 UTC m=+22.539845518" watchObservedRunningTime="2026-07-24 17:40:31.423066476 +0000 UTC m=+22.539993816"
Jul 24 17:40:31.440888 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:31.440834    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lc894" podStartSLOduration=3.73791102 podStartE2EDuration="22.440813903s" totalImagesPullingTime="18.702902883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:31.440419093 +0000 UTC m=+22.557346450" watchObservedRunningTime="2026-07-24 17:40:31.440813903 +0000 UTC m=+22.557741245"
Jul 24 17:40:32.294147 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.294120    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:32.294325 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:32.294247    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:32.294325 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.294308    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:32.295191 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:32.294451    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:32.327076 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.326620    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 17:40:32.386103 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386077    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:40:32.386477 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386450    2551 generic.go:356] "Generic (PLEG): container finished" podID="37e312f0-8531-4f45-9267-2902b33b0788" containerID="a3a0fd2adfe6754a6d80835f015a1aa50db02cecc232b54fa685db7c4717acb4" exitCode=1
Jul 24 17:40:32.386614 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386515    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"04c40a82ef62adbb8b65e925c58f85d8b162010ddb8ca212125d71aa50b79d4e"}
Jul 24 17:40:32.386614 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386602    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"b99d42e82ca59ae6121ff773054c711cb5e4eefd0d8e6dc7c2ce0cb04ce629e7"}
Jul 24 17:40:32.386729 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386654    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"f0b16a6c4e1754481ac65b8e29f714b2db4048d74fd4276b5e68a8a7d76c3cc9"}
Jul 24 17:40:32.386729 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386704    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"6de43b77982620266a6c492e328d2b1e792cc9dee96876cca816ed792435e3b1"}
Jul 24 17:40:32.386877 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386750    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerDied","Data":"a3a0fd2adfe6754a6d80835f015a1aa50db02cecc232b54fa685db7c4717acb4"}
Jul 24 17:40:32.386877 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.386797    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"089c6a0a57ceb1bd0310f45b9e12c003e05374487cd492790a8d023504c7730f"}
Jul 24 17:40:32.388709 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.388684    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59" event={"ID":"0dc17ebe-0652-4923-a6f2-14934a79ac61","Type":"ContainerStarted","Data":"cde03d2b0e82447273e0685bd31f464b41b91489c680842226b612e0a9b11e4c"}
Jul 24 17:40:32.390291 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.390266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qt655" event={"ID":"e0bbfe33-71c7-4c5a-b361-b9d2c4b6ed22","Type":"ContainerStarted","Data":"844f9d0cf0c81342854f13d8a1e3d624a6648d5316664d738775eede6dddf707"}
Jul 24 17:40:32.408455 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:32.408407    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n5qts" podStartSLOduration=4.675701537 podStartE2EDuration="23.408365846s" totalImagesPullingTime="18.732664309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:31.487801789 +0000 UTC m=+22.604729131" watchObservedRunningTime="2026-07-24 17:40:32.408365846 +0000 UTC m=+23.525293185"
Jul 24 17:40:33.225998 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:33.225895    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:40:32.32664236Z","UUID":"7337799e-a0ee-4c8c-98b2-937bbf026a7b","Handler":null,"Name":"","Endpoint":""}
Jul 24 17:40:33.227468 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:33.227444    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 17:40:33.227604 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:33.227521    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 17:40:33.294888 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:33.294845    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:33.295344 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:33.295036    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:34.294883 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:34.294854    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:34.295032 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:34.294854    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:34.295032 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:34.294965    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:34.295316 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:34.295025    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:34.397634 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:34.397589    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59" event={"ID":"0dc17ebe-0652-4923-a6f2-14934a79ac61","Type":"ContainerStarted","Data":"e183b463ad888b0cd1c8685f23a4098b010e31dcb1bc7647b68e0b330da7c874"}
Jul 24 17:40:34.423685 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:34.423640    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qt655" podStartSLOduration=6.711557136 podStartE2EDuration="25.423623454s" totalImagesPullingTime="18.712066318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:32.405234848 +0000 UTC m=+23.522162189" watchObservedRunningTime="2026-07-24 17:40:34.423623454 +0000 UTC m=+25.540550794"
Jul 24 17:40:35.294742 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:35.294564    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:35.294919 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:35.294843    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:36.287434 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.287405    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:36.287965 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.287950    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:36.294262 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.294243    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:36.294401 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:36.294352    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:36.294474 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.294404    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:36.294526 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:36.294488    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:36.303015 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.302972    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdm59" podStartSLOduration=6.283861026 podStartE2EDuration="27.302959478s" totalImagesPullingTime="21.019098452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:34.424416979 +0000 UTC m=+25.541344320" watchObservedRunningTime="2026-07-24 17:40:36.302959478 +0000 UTC m=+27.419886883"
Jul 24 17:40:36.405042 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.405022    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:40:36.405403 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.405359    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"5998d20dd12b336de4135223fc9771a349e3a76556a2e8286099e960cf3f9528"}
Jul 24 17:40:36.407299 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.407276    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerStarted","Data":"9a8d12aa44af49066a5d4fe1344c79c4b0ad485cc5b0a7475a081889a49cf9f8"}
Jul 24 17:40:36.407492 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.407475    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:36.407902 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:36.407889    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q2fwc"
Jul 24 17:40:37.294616 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:37.294587    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:37.295135 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:37.294725    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:37.411155 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:37.411080    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521" containerID="9a8d12aa44af49066a5d4fe1344c79c4b0ad485cc5b0a7475a081889a49cf9f8" exitCode=0
Jul 24 17:40:37.411287 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:37.411148    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerDied","Data":"9a8d12aa44af49066a5d4fe1344c79c4b0ad485cc5b0a7475a081889a49cf9f8"}
Jul 24 17:40:38.294440 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.294253    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:38.294586 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.294283    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:38.294586 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:38.294551    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:38.295060 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:38.294599    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:38.415242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.415216    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521" containerID="a69d9c5cd63933a716011ffe5ce1060f4250c7d9c2126d23d4c9b2ba4f1f4de7" exitCode=0
Jul 24 17:40:38.415429 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.415294    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerDied","Data":"a69d9c5cd63933a716011ffe5ce1060f4250c7d9c2126d23d4c9b2ba4f1f4de7"}
Jul 24 17:40:38.419620 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.419595    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:40:38.421783 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.421754    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"1d4d2c912a06e3875364fe8445480a31cb95aafe9d33ebfbcf2ea17db48a20b5"}
Jul 24 17:40:38.422753 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.422093    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:38.422753 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.422150    2551 scope.go:117] "RemoveContainer" containerID="a3a0fd2adfe6754a6d80835f015a1aa50db02cecc232b54fa685db7c4717acb4"
Jul 24 17:40:38.422753 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.422266    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:38.423072 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.423052    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:38.440282 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.440261    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:38.440406 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:38.440395    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:40:39.295191 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:39.295155    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:39.295668 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:39.295285    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:39.427909 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:39.427884    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:40:39.428315 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:39.428290    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" event={"ID":"37e312f0-8531-4f45-9267-2902b33b0788","Type":"ContainerStarted","Data":"5f2d3828943bb93686a94bdd0b97023d50cae9f081ee41d2bbd6afa2b2b1b3d7"}
Jul 24 17:40:39.779147 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:39.779115    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84cpg_1b44312a-d41c-4158-addc-4fd4b7a0c61f/dns-node-resolver/0.log"
Jul 24 17:40:40.294281 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.294251    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:40.294460 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.294300    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:40.294460 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:40.294392    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:40.294460 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:40.294443    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:40.433036 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.432999    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521" containerID="6f5d3f4d95430c8a28a6f976aee4c85961e05ef7a87f83f508984050d31525ee" exitCode=0
Jul 24 17:40:40.433443 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.433083    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerDied","Data":"6f5d3f4d95430c8a28a6f976aee4c85961e05ef7a87f83f508984050d31525ee"}
Jul 24 17:40:40.454831 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.454792    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht" podStartSLOduration=12.252500343 podStartE2EDuration="31.454777745s" totalImagesPullingTime="19.202277402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:39.474458618 +0000 UTC m=+30.591385958" watchObservedRunningTime="2026-07-24 17:40:40.454777745 +0000 UTC m=+31.571705085"
Jul 24 17:40:40.502285 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.502253    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5gq5"]
Jul 24 17:40:40.502471 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.502368    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:40.502540 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:40.502516    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:40.506230 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.506204    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-46rhf"]
Jul 24 17:40:40.506437 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.506422    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wnsct"]
Jul 24 17:40:40.506642 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.506620    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:40.506764 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:40.506737    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:40.506940 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.506922    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:40.507028 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:40.507011    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:40.952316 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:40.952288    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5x8nc_38c17424-6bce-4314-a59d-e0034753af29/node-ca/0.log"
Jul 24 17:40:42.294447 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:42.294254    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:42.294816 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:42.294255    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:42.294816 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:42.294516    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:42.294816 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:42.294634    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:42.294816 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:42.294258    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:42.294816 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:42.294711    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:42.930334 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:42.930298    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:42.930553 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:42.930485    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:42.930628 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:42.930568    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret podName:d346c91f-4715-48f3-873f-a9cb78c34f74 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:14.930548086 +0000 UTC m=+66.047475411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret") pod "global-pull-secret-syncer-f5gq5" (UID: "d346c91f-4715-48f3-873f-a9cb78c34f74") : object "kube-system"/"original-pull-secret" not registered
Jul 24 17:40:43.031738 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:43.031702    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:43.031738 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:43.031744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:43.031951 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:43.031870    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 17:40:43.031951 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:43.031874    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:43.031951 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:43.031892    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 17:40:43.031951 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:43.031904    2551 projected.go:196] Error preparing data for projected volume kube-api-access-72d97 for pod openshift-network-diagnostics/network-check-target-46rhf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:43.032092 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:43.031957    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs podName:ca6bd76d-63fe-4c44-a496-6f9ab029b2cb nodeName:}" failed. No retries permitted until 2026-07-24 17:41:15.031933706 +0000 UTC m=+66.148861030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs") pod "network-metrics-daemon-wnsct" (UID: "ca6bd76d-63fe-4c44-a496-6f9ab029b2cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 17:40:43.032092 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:43.031977    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97 podName:86dfdc0f-5604-4754-9a47-65b0d7c910fb nodeName:}" failed. No retries permitted until 2026-07-24 17:41:15.031966728 +0000 UTC m=+66.148894052 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-72d97" (UniqueName: "kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97") pod "network-check-target-46rhf" (UID: "86dfdc0f-5604-4754-9a47-65b0d7c910fb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 17:40:44.294586 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:44.294552    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:44.294586 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:44.294586    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:44.295065 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:44.294687    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:44.295065 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:44.294775    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:44.295065 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:44.294817    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:44.295065 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:44.294910    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:45.448536 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:45.448504    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521" containerID="23bd2c25b48deadc8f18fbd11d44cc07cba5201fd25fe2aa25774ef916c9b32e" exitCode=0
Jul 24 17:40:45.449068 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:45.448565    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerDied","Data":"23bd2c25b48deadc8f18fbd11d44cc07cba5201fd25fe2aa25774ef916c9b32e"}
Jul 24 17:40:46.294545 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:46.294514    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:46.294712 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:46.294549    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:46.294712 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:46.294598    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:46.294712 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:46.294611    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:46.294830 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:46.294708    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:46.294830 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:46.294789    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:46.453488 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:46.453452    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521" containerID="3a6984b8ea6e56c277437ada53dc052973d1d1c3171c79333d3bee4da4eec125" exitCode=0
Jul 24 17:40:46.453873 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:46.453494    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerDied","Data":"3a6984b8ea6e56c277437ada53dc052973d1d1c3171c79333d3bee4da4eec125"}
Jul 24 17:40:47.458430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:47.458399    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bnscl" event={"ID":"2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521","Type":"ContainerStarted","Data":"6752fff96a6bbe211f86b9570442b08816f8b8ef5bc531c3eb50a07749d6dccf"}
Jul 24 17:40:47.479522 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:47.479480    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bnscl" podStartSLOduration=8.692423802 podStartE2EDuration="38.479466348s" totalImagesPullingTime="29.787042546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:47.478627291 +0000 UTC m=+38.595554631" watchObservedRunningTime="2026-07-24 17:40:47.479466348 +0000 UTC m=+38.596393719"
Jul 24 17:40:48.294303 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:48.294270    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:48.294484 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:48.294272    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:48.294484 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:48.294370    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:48.294484 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:48.294279    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:48.294484 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:48.294465    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:48.294617 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:48.294563    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:50.294434 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:50.294355    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:50.294879 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:50.294463    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:50.294879 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:50.294471    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:50.294879 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:50.294488    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:50.294879 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:50.294559    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:50.294879 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:50.294625    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:52.294134 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:52.294092    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:52.294557 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:52.294194    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:52.294557 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:52.294205    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:52.294557 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:52.294200    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:52.294557 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:52.294277    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:52.294557 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:52.294418    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:54.294837 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:54.294795    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:54.295229 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:54.294848    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:54.295229 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:54.294848    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:54.295229 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:54.294905    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:54.295229 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:54.295017    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:54.295229 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:54.295065    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:56.294692 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:56.294658    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:56.295144 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:56.294770    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:56.295144 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:56.294781    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:56.295144 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:56.294781    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:40:56.295144 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:56.294857    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:56.295144 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:56.294949    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:58.294739 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:58.294701    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:40:58.294739 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:58.294741    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:40:58.295144 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:40:58.294741    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:40:58.295144 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:58.294811    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:40:58.295144 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:58.294895    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:40:58.295144 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:40:58.294961    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:41:00.294102 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:00.294061    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:00.294521 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:00.294109    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:41:00.294521 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:00.294160    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:41:00.294521 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:00.294174    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:41:00.294521 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:00.294238    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:41:00.294521 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:00.294295    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:41:02.294700 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:02.294663    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:41:02.295231 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:02.294760    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:41:02.295231 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:02.294765    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:41:02.295231 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:02.294770    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:02.295231 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:02.294900    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:41:02.295231 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:02.294850    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:41:04.294681 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.294649    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:04.295037 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.294745    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:41:04.295037 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:04.294752    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46rhf" podUID="86dfdc0f-5604-4754-9a47-65b0d7c910fb"
Jul 24 17:41:04.295037 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.294775    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:41:04.295037 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:04.294843    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f5gq5" podUID="d346c91f-4715-48f3-873f-a9cb78c34f74"
Jul 24 17:41:04.295037 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:04.294909    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wnsct" podUID="ca6bd76d-63fe-4c44-a496-6f9ab029b2cb"
Jul 24 17:41:04.711053 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.711027    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-87.ec2.internal" event="NodeReady"
Jul 24 17:41:04.711211 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.711139    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 17:41:04.754097 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.754067    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5c67v"]
Jul 24 17:41:04.755692 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.755677    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.757593 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.757572    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rc7j7"]
Jul 24 17:41:04.758690 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.758670    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tq4qz\""
Jul 24 17:41:04.758690 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.758685    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 17:41:04.758808 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.758717    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 17:41:04.759165 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.759150    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.761207 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.761191    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 17:41:04.761458 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.761442    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 17:41:04.761533 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.761454    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9gf7\""
Jul 24 17:41:04.761533 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.761496    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 17:41:04.761653 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.761539    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 17:41:04.766666 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.766646    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c67v"]
Jul 24 17:41:04.773428 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.773408    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rc7j7"]
Jul 24 17:41:04.788041 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788019    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00eca871-94cf-449c-aac1-ec4971d7a909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.788148 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788048    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2879b37-cd85-4cbf-9899-657d371e03b0-metrics-tls\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.788148 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788068    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4g6d\" (UniqueName: \"kubernetes.io/projected/f2879b37-cd85-4cbf-9899-657d371e03b0-kube-api-access-k4g6d\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.788148 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2879b37-cd85-4cbf-9899-657d371e03b0-tmp-dir\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.788148 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788142    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00eca871-94cf-449c-aac1-ec4971d7a909-crio-socket\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.788277 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788186    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00eca871-94cf-449c-aac1-ec4971d7a909-data-volume\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.788277 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788203    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00eca871-94cf-449c-aac1-ec4971d7a909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.788277 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788219    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9r6\" (UniqueName: \"kubernetes.io/projected/00eca871-94cf-449c-aac1-ec4971d7a909-kube-api-access-8b9r6\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.788277 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.788253    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2879b37-cd85-4cbf-9899-657d371e03b0-config-volume\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.856313 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.856283    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kfr4f"]
Jul 24 17:41:04.857868 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.857852    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:04.860689 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.860666    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 17:41:04.861341 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.861068    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 17:41:04.861470 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.861449    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jpmgf\""
Jul 24 17:41:04.861533 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.861467    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 17:41:04.867638 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.867611    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kfr4f"]
Jul 24 17:41:04.888991 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.888960    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00eca871-94cf-449c-aac1-ec4971d7a909-data-volume\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.888991 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.888989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00eca871-94cf-449c-aac1-ec4971d7a909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.889142 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889007    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9r6\" (UniqueName: \"kubernetes.io/projected/00eca871-94cf-449c-aac1-ec4971d7a909-kube-api-access-8b9r6\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.889185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889143    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2879b37-cd85-4cbf-9899-657d371e03b0-config-volume\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.889185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889177    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dn9b\" (UniqueName: \"kubernetes.io/projected/361a1a91-150a-472a-9950-49eafdb49c3b-kube-api-access-7dn9b\") pod \"ingress-canary-kfr4f\" (UID: \"361a1a91-150a-472a-9950-49eafdb49c3b\") " pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:04.889272 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889232    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00eca871-94cf-449c-aac1-ec4971d7a909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.889272 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889260    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2879b37-cd85-4cbf-9899-657d371e03b0-metrics-tls\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.889355 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889289    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4g6d\" (UniqueName: \"kubernetes.io/projected/f2879b37-cd85-4cbf-9899-657d371e03b0-kube-api-access-k4g6d\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.889355 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889315    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00eca871-94cf-449c-aac1-ec4971d7a909-data-volume\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.889355 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889324    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2879b37-cd85-4cbf-9899-657d371e03b0-tmp-dir\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.889526 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889404    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00eca871-94cf-449c-aac1-ec4971d7a909-crio-socket\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.889526 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889437    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/361a1a91-150a-472a-9950-49eafdb49c3b-cert\") pod \"ingress-canary-kfr4f\" (UID: \"361a1a91-150a-472a-9950-49eafdb49c3b\") " pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:04.889526 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889516    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00eca871-94cf-449c-aac1-ec4971d7a909-crio-socket\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.889680 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889589    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2879b37-cd85-4cbf-9899-657d371e03b0-tmp-dir\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.889811 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889793    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00eca871-94cf-449c-aac1-ec4971d7a909-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.889883 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.889809    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2879b37-cd85-4cbf-9899-657d371e03b0-config-volume\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.893135 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.893118    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2879b37-cd85-4cbf-9899-657d371e03b0-metrics-tls\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.893179 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.893146    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00eca871-94cf-449c-aac1-ec4971d7a909-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.896546 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.896523    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4g6d\" (UniqueName: \"kubernetes.io/projected/f2879b37-cd85-4cbf-9899-657d371e03b0-kube-api-access-k4g6d\") pod \"dns-default-5c67v\" (UID: \"f2879b37-cd85-4cbf-9899-657d371e03b0\") " pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:04.897492 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.897470    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9r6\" (UniqueName: \"kubernetes.io/projected/00eca871-94cf-449c-aac1-ec4971d7a909-kube-api-access-8b9r6\") pod \"insights-runtime-extractor-rc7j7\" (UID: \"00eca871-94cf-449c-aac1-ec4971d7a909\") " pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:04.990243 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.990155    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/361a1a91-150a-472a-9950-49eafdb49c3b-cert\") pod \"ingress-canary-kfr4f\" (UID: \"361a1a91-150a-472a-9950-49eafdb49c3b\") " pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:04.990243 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.990219    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dn9b\" (UniqueName: \"kubernetes.io/projected/361a1a91-150a-472a-9950-49eafdb49c3b-kube-api-access-7dn9b\") pod \"ingress-canary-kfr4f\" (UID: \"361a1a91-150a-472a-9950-49eafdb49c3b\") " pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:04.992726 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.992706    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/361a1a91-150a-472a-9950-49eafdb49c3b-cert\") pod \"ingress-canary-kfr4f\" (UID: \"361a1a91-150a-472a-9950-49eafdb49c3b\") " pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:04.998853 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:04.998826    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dn9b\" (UniqueName: \"kubernetes.io/projected/361a1a91-150a-472a-9950-49eafdb49c3b-kube-api-access-7dn9b\") pod \"ingress-canary-kfr4f\" (UID: \"361a1a91-150a-472a-9950-49eafdb49c3b\") " pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:05.066587 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.066554    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:05.071811 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.071794    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rc7j7"
Jul 24 17:41:05.169884 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.169851    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kfr4f"
Jul 24 17:41:05.205950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.205920    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rc7j7"]
Jul 24 17:41:05.209259 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:05.209228    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00eca871_94cf_449c_aac1_ec4971d7a909.slice/crio-b3e7058f06843dd1f98b63d3fa34a0cccdcb2adf035c01de4dd6159d9a6ff9c1 WatchSource:0}: Error finding container b3e7058f06843dd1f98b63d3fa34a0cccdcb2adf035c01de4dd6159d9a6ff9c1: Status 404 returned error can't find the container with id b3e7058f06843dd1f98b63d3fa34a0cccdcb2adf035c01de4dd6159d9a6ff9c1
Jul 24 17:41:05.225965 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.225880    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c67v"]
Jul 24 17:41:05.229357 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:05.229324    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2879b37_cd85_4cbf_9899_657d371e03b0.slice/crio-17131fd0411a16bccb482a29a131dcc9ef09b80862d91bbff95ac97786733f4f WatchSource:0}: Error finding container 17131fd0411a16bccb482a29a131dcc9ef09b80862d91bbff95ac97786733f4f: Status 404 returned error can't find the container with id 17131fd0411a16bccb482a29a131dcc9ef09b80862d91bbff95ac97786733f4f
Jul 24 17:41:05.297781 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.297752    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kfr4f"]
Jul 24 17:41:05.301032 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:05.301008    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361a1a91_150a_472a_9950_49eafdb49c3b.slice/crio-2471481d1649f0f6aa4072ec68d67a5b34d6105f34e7550ff6c29e7b04e5c54e WatchSource:0}: Error finding container 2471481d1649f0f6aa4072ec68d67a5b34d6105f34e7550ff6c29e7b04e5c54e: Status 404 returned error can't find the container with id 2471481d1649f0f6aa4072ec68d67a5b34d6105f34e7550ff6c29e7b04e5c54e
Jul 24 17:41:05.497913 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.497876    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kfr4f" event={"ID":"361a1a91-150a-472a-9950-49eafdb49c3b","Type":"ContainerStarted","Data":"2471481d1649f0f6aa4072ec68d67a5b34d6105f34e7550ff6c29e7b04e5c54e"}
Jul 24 17:41:05.499032 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.498998    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rc7j7" event={"ID":"00eca871-94cf-449c-aac1-ec4971d7a909","Type":"ContainerStarted","Data":"b3e7058f06843dd1f98b63d3fa34a0cccdcb2adf035c01de4dd6159d9a6ff9c1"}
Jul 24 17:41:05.500056 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:05.500031    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c67v" event={"ID":"f2879b37-cd85-4cbf-9899-657d371e03b0","Type":"ContainerStarted","Data":"17131fd0411a16bccb482a29a131dcc9ef09b80862d91bbff95ac97786733f4f"}
Jul 24 17:41:06.294319 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.294210    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:41:06.294319 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.294252    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:06.294319 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.294256    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:41:06.297514 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.297490    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:41:06.297656 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.297633    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:41:06.297888 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.297851    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t8nmh\""
Jul 24 17:41:06.297888 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.297866    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:41:06.298284 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.298070    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:41:06.298284 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:06.298238    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-62mvz\""
Jul 24 17:41:09.663221 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.663149    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-fph7j"]
Jul 24 17:41:09.876834 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.876796    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-fph7j"]
Jul 24 17:41:09.877072 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.876956    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:09.879476 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.879448    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 17:41:09.880281 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.880254    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 17:41:09.880464 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.880369    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q4szt\""
Jul 24 17:41:09.880464 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.880255    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 17:41:09.880618 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.880602    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 17:41:09.880675 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.880642    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 17:41:09.934972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.934884    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e289892c-3e93-4479-a726-c9ff4896b0d2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:09.934972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.934918    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j97p\" (UniqueName: \"kubernetes.io/projected/e289892c-3e93-4479-a726-c9ff4896b0d2-kube-api-access-7j97p\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:09.934972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.934942    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e289892c-3e93-4479-a726-c9ff4896b0d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:09.935242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:09.935016    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e289892c-3e93-4479-a726-c9ff4896b0d2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.035347 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.035315    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e289892c-3e93-4479-a726-c9ff4896b0d2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.035347 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.035346    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j97p\" (UniqueName: \"kubernetes.io/projected/e289892c-3e93-4479-a726-c9ff4896b0d2-kube-api-access-7j97p\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.035540 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.035369    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e289892c-3e93-4479-a726-c9ff4896b0d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.035540 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.035410    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e289892c-3e93-4479-a726-c9ff4896b0d2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.036241 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.036140    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e289892c-3e93-4479-a726-c9ff4896b0d2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.037949 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.037923    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e289892c-3e93-4479-a726-c9ff4896b0d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.038046 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.037963    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e289892c-3e93-4479-a726-c9ff4896b0d2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.044524 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.044496    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j97p\" (UniqueName: \"kubernetes.io/projected/e289892c-3e93-4479-a726-c9ff4896b0d2-kube-api-access-7j97p\") pod \"prometheus-operator-84d69f794d-fph7j\" (UID: \"e289892c-3e93-4479-a726-c9ff4896b0d2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.187840 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.187770    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j"
Jul 24 17:41:10.444539 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:10.444445    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2hht"
Jul 24 17:41:11.155139 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.155010    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-fph7j"]
Jul 24 17:41:11.519640 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.519583    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kfr4f" event={"ID":"361a1a91-150a-472a-9950-49eafdb49c3b","Type":"ContainerStarted","Data":"3bfed176ef75e6217150a2cb69f8960b19f0a67c1e25da88d9b2512660230cee"}
Jul 24 17:41:11.521876 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.521848    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c67v" event={"ID":"f2879b37-cd85-4cbf-9899-657d371e03b0","Type":"ContainerStarted","Data":"87009731fbaa554b84f2e2199d8f9c4bf9d384a236ab535597c926bff393f966"}
Jul 24 17:41:11.522008 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.521901    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c67v" event={"ID":"f2879b37-cd85-4cbf-9899-657d371e03b0","Type":"ContainerStarted","Data":"c601aae110bf074596e0953831dae5fafcedbdbb52c11915491e900a427cfb58"}
Jul 24 17:41:11.522008 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.521997    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:11.523339 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.523313    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j" event={"ID":"e289892c-3e93-4479-a726-c9ff4896b0d2","Type":"ContainerStarted","Data":"ccfcf5622161604e9eea1daf16bf2238d81ccd145d0fa38caa473ee23fe2b7b2"}
Jul 24 17:41:11.525007 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.524984    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rc7j7" event={"ID":"00eca871-94cf-449c-aac1-ec4971d7a909","Type":"ContainerStarted","Data":"ecabfcc98fdca054f395a3713943cc87ae10d9c2d7f7933e94aac3172f9b0618"}
Jul 24 17:41:11.537802 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.537726    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kfr4f" podStartSLOduration=1.8509162940000001 podStartE2EDuration="7.537710727s" totalImagesPullingTime="5.686794433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:11.537331661 +0000 UTC m=+62.654258999" watchObservedRunningTime="2026-07-24 17:41:11.537710727 +0000 UTC m=+62.654638069"
Jul 24 17:41:11.554360 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:11.554312    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5c67v" podStartSLOduration=2.12268645 podStartE2EDuration="7.554297093s" totalImagesPullingTime="5.431610643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:11.55361612 +0000 UTC m=+62.670543460" watchObservedRunningTime="2026-07-24 17:41:11.554297093 +0000 UTC m=+62.671224434"
Jul 24 17:41:13.533229 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:13.533197    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j" event={"ID":"e289892c-3e93-4479-a726-c9ff4896b0d2","Type":"ContainerStarted","Data":"552248fb698340d2b0603b87dd0f791f8a8b456140070fb0c38714b97217e7f1"}
Jul 24 17:41:13.533733 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:13.533266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j" event={"ID":"e289892c-3e93-4479-a726-c9ff4896b0d2","Type":"ContainerStarted","Data":"f304e31a0e2dff4fea9a47abb21a8119e368e88cfc51d508f03d9901fa215421"}
Jul 24 17:41:13.550044 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:13.549988    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-fph7j" podStartSLOduration=3.160568508 podStartE2EDuration="4.549972666s" totalImagesPullingTime="1.389404158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:13.548931417 +0000 UTC m=+64.665858758" watchObservedRunningTime="2026-07-24 17:41:13.549972666 +0000 UTC m=+64.666899997"
Jul 24 17:41:14.976870 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:14.976829    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:41:14.982197 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:14.982174    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 17:41:14.989240 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:14.989223    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d346c91f-4715-48f3-873f-a9cb78c34f74-original-pull-secret\") pod \"global-pull-secret-syncer-f5gq5\" (UID: \"d346c91f-4715-48f3-873f-a9cb78c34f74\") " pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:41:15.012300 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.012234    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"]
Jul 24 17:41:15.015341 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.015326    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.017325 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.017297    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 17:41:15.017449 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.017309    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f6qwx\""
Jul 24 17:41:15.017449 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.017426    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 17:41:15.023187 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.023165    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"]
Jul 24 17:41:15.025282 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.025263    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f5gq5"
Jul 24 17:41:15.028393 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.028353    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4bwj9"]
Jul 24 17:41:15.031855 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.031834    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.032513 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.032494    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"]
Jul 24 17:41:15.033955 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.033933    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 17:41:15.034221 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.034204    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 17:41:15.034290 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.034259    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sft4t\""
Jul 24 17:41:15.034610 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.034593    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 17:41:15.035733 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.035716    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.038934 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.038915    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 17:41:15.039043 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.038957    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 17:41:15.039043 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.038988    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-n8r4j\""
Jul 24 17:41:15.039364 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.039349    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 17:41:15.050208 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.050188    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"]
Jul 24 17:41:15.078224 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078193    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2bbb19c-ed35-4308-a300-75b0e4021dd8-metrics-client-ca\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.078430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078239    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.078430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078272    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-wtmp\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.078430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078315    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-sys\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.078430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.078430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078406    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.078430 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078427    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5fbc\" (UniqueName: \"kubernetes.io/projected/f02594fb-33a4-432f-84c4-1937c3077ccb-kube-api-access-s5fbc\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078448    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7hs\" (UniqueName: \"kubernetes.io/projected/c2bbb19c-ed35-4308-a300-75b0e4021dd8-kube-api-access-5z7hs\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078495    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078521    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-root\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078543    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-accelerators-collector-config\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078579    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078665    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078746    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-textfile\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.078777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078776    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.079131 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078810    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6ng\" (UniqueName: \"kubernetes.io/projected/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-api-access-zk6ng\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.079131 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078841    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-tls\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.079131 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078867    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.079131 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078892    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02594fb-33a4-432f-84c4-1937c3077ccb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.079131 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078915    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73df4cb3-0d16-488a-a859-4d1a5de021d2-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.079131 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.078939    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73df4cb3-0d16-488a-a859-4d1a5de021d2-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.083780 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.083756    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 17:41:15.084661 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.084641    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 17:41:15.091516 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.091458    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 17:41:15.092956 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.092679    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca6bd76d-63fe-4c44-a496-6f9ab029b2cb-metrics-certs\") pod \"network-metrics-daemon-wnsct\" (UID: \"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb\") " pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:41:15.101801 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.101772    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72d97\" (UniqueName: \"kubernetes.io/projected/86dfdc0f-5604-4754-9a47-65b0d7c910fb-kube-api-access-72d97\") pod \"network-check-target-46rhf\" (UID: \"86dfdc0f-5604-4754-9a47-65b0d7c910fb\") " pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:15.154578 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.154509    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f5gq5"]
Jul 24 17:41:15.158730 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:15.158682    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd346c91f_4715_48f3_873f_a9cb78c34f74.slice/crio-fd53456805ab846e108435c4270b7cea6a27efb92583e4308133e9ced3a0768e WatchSource:0}: Error finding container fd53456805ab846e108435c4270b7cea6a27efb92583e4308133e9ced3a0768e: Status 404 returned error can't find the container with id fd53456805ab846e108435c4270b7cea6a27efb92583e4308133e9ced3a0768e
Jul 24 17:41:15.180005 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.179977    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2bbb19c-ed35-4308-a300-75b0e4021dd8-metrics-client-ca\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180125 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180012    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.180125 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180029    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-wtmp\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180125 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180049    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-sys\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180125 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180075    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.180125 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180099    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.180125 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5fbc\" (UniqueName: \"kubernetes.io/projected/f02594fb-33a4-432f-84c4-1937c3077ccb-kube-api-access-s5fbc\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7hs\" (UniqueName: \"kubernetes.io/projected/c2bbb19c-ed35-4308-a300-75b0e4021dd8-kube-api-access-5z7hs\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:15.180154    2551 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180180    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-wtmp\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:15.180217    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-tls podName:73df4cb3-0d16-488a-a859-4d1a5de021d2 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:15.680194734 +0000 UTC m=+66.797122053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-nfmll" (UID: "73df4cb3-0d16-488a-a859-4d1a5de021d2") : secret "kube-state-metrics-tls" not found
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180249    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-root\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180282    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-accelerators-collector-config\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180331    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-textfile\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180355    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180149    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-sys\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180459 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180419    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6ng\" (UniqueName: \"kubernetes.io/projected/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-api-access-zk6ng\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180497    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-tls\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:15.180504    2551 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180524    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:15.180556    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-tls podName:f02594fb-33a4-432f-84c4-1937c3077ccb nodeName:}" failed. No retries permitted until 2026-07-24 17:41:15.680539466 +0000 UTC m=+66.797466806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-pxsnj" (UID: "f02594fb-33a4-432f-84c4-1937c3077ccb") : secret "openshift-state-metrics-tls" not found
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180585    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02594fb-33a4-432f-84c4-1937c3077ccb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180604    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73df4cb3-0d16-488a-a859-4d1a5de021d2-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180620    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73df4cb3-0d16-488a-a859-4d1a5de021d2-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180712    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2bbb19c-ed35-4308-a300-75b0e4021dd8-metrics-client-ca\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-textfile\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.180972 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.180932    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-accelerators-collector-config\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.181491 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:15.180999    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 17:41:15.181491 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:15.181080    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-tls podName:c2bbb19c-ed35-4308-a300-75b0e4021dd8 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:15.681063085 +0000 UTC m=+66.797990417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-tls") pod "node-exporter-4bwj9" (UID: "c2bbb19c-ed35-4308-a300-75b0e4021dd8") : secret "node-exporter-tls" not found
Jul 24 17:41:15.181491 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.181135    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c2bbb19c-ed35-4308-a300-75b0e4021dd8-root\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.181491 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.181185    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73df4cb3-0d16-488a-a859-4d1a5de021d2-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.181491 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.181473    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73df4cb3-0d16-488a-a859-4d1a5de021d2-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.181760 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.181561    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02594fb-33a4-432f-84c4-1937c3077ccb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.181810 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.181772    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.182894 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.182874    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.183604 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.183577    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.183684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.183658    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.191557 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.191538    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7hs\" (UniqueName: \"kubernetes.io/projected/c2bbb19c-ed35-4308-a300-75b0e4021dd8-kube-api-access-5z7hs\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.191883 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.191839    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6ng\" (UniqueName: \"kubernetes.io/projected/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-api-access-zk6ng\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.191883 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.191849    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5fbc\" (UniqueName: \"kubernetes.io/projected/f02594fb-33a4-432f-84c4-1937c3077ccb-kube-api-access-s5fbc\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.312152 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.312108    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-62mvz\""
Jul 24 17:41:15.319962 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.319733    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wnsct"
Jul 24 17:41:15.321812 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.321790    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t8nmh\""
Jul 24 17:41:15.330288 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.330258    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:15.474082 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.474049    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wnsct"]
Jul 24 17:41:15.478536 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:15.478489    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6bd76d_63fe_4c44_a496_6f9ab029b2cb.slice/crio-9ea0bcb31ad5c5e6c1748ab48d3a613197e1b210956d946bb5f1d89a626f3b07 WatchSource:0}: Error finding container 9ea0bcb31ad5c5e6c1748ab48d3a613197e1b210956d946bb5f1d89a626f3b07: Status 404 returned error can't find the container with id 9ea0bcb31ad5c5e6c1748ab48d3a613197e1b210956d946bb5f1d89a626f3b07
Jul 24 17:41:15.493293 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.493268    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-46rhf"]
Jul 24 17:41:15.498031 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:15.498002    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dfdc0f_5604_4754_9a47_65b0d7c910fb.slice/crio-be63e824abad55f8b031f2f3450f7d960c2b33865d3485a0e262f1de01086c56 WatchSource:0}: Error finding container be63e824abad55f8b031f2f3450f7d960c2b33865d3485a0e262f1de01086c56: Status 404 returned error can't find the container with id be63e824abad55f8b031f2f3450f7d960c2b33865d3485a0e262f1de01086c56
Jul 24 17:41:15.546367 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.546329    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rc7j7" event={"ID":"00eca871-94cf-449c-aac1-ec4971d7a909","Type":"ContainerStarted","Data":"64220977f83408c8b0640b37aeb50e3768e77f689de0cdd63261a1a2368ce510"}
Jul 24 17:41:15.547621 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.547589    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-46rhf" event={"ID":"86dfdc0f-5604-4754-9a47-65b0d7c910fb","Type":"ContainerStarted","Data":"be63e824abad55f8b031f2f3450f7d960c2b33865d3485a0e262f1de01086c56"}
Jul 24 17:41:15.548734 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.548705    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wnsct" event={"ID":"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb","Type":"ContainerStarted","Data":"9ea0bcb31ad5c5e6c1748ab48d3a613197e1b210956d946bb5f1d89a626f3b07"}
Jul 24 17:41:15.549920 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.549898    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5gq5" event={"ID":"d346c91f-4715-48f3-873f-a9cb78c34f74","Type":"ContainerStarted","Data":"fd53456805ab846e108435c4270b7cea6a27efb92583e4308133e9ced3a0768e"}
Jul 24 17:41:15.688087 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.688001    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-tls\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.688087 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.688069    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.688304 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.688096    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.691112 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.691081    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f02594fb-33a4-432f-84c4-1937c3077ccb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pxsnj\" (UID: \"f02594fb-33a4-432f-84c4-1937c3077ccb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.691363 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.691336    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c2bbb19c-ed35-4308-a300-75b0e4021dd8-node-exporter-tls\") pod \"node-exporter-4bwj9\" (UID: \"c2bbb19c-ed35-4308-a300-75b0e4021dd8\") " pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.691475 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.691432    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73df4cb3-0d16-488a-a859-4d1a5de021d2-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-nfmll\" (UID: \"73df4cb3-0d16-488a-a859-4d1a5de021d2\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.925491 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.925441    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"
Jul 24 17:41:15.952763 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.952551    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4bwj9"
Jul 24 17:41:15.958068 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:15.957998    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"
Jul 24 17:41:15.993016 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:15.992857    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bbb19c_ed35_4308_a300_75b0e4021dd8.slice/crio-e8d8738c9b91351045582c85683452436c3bcbbd058a4573ecfad45fddcfcdbe WatchSource:0}: Error finding container e8d8738c9b91351045582c85683452436c3bcbbd058a4573ecfad45fddcfcdbe: Status 404 returned error can't find the container with id e8d8738c9b91351045582c85683452436c3bcbbd058a4573ecfad45fddcfcdbe
Jul 24 17:41:16.093774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.088035    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:41:16.120778 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.120224    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:41:16.120778 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.120429    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.124672 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.123548    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 17:41:16.124672 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.123957    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 17:41:16.124672 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.124170    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pq2mq\""
Jul 24 17:41:16.124672 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.124361    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 17:41:16.124672 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.124668    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 17:41:16.124999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.124726    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 17:41:16.124999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.124892    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 17:41:16.124999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.124937    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 17:41:16.125159 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.124892    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 17:41:16.125159 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.125124    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 17:41:16.148135 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.148045    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj"]
Jul 24 17:41:16.159983 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:16.159901    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02594fb_33a4_432f_84c4_1937c3077ccb.slice/crio-46d9f8ef66f29e5af98c8e87328f117a5bc0a1af39dd5c7c758972bf41aad954 WatchSource:0}: Error finding container 46d9f8ef66f29e5af98c8e87328f117a5bc0a1af39dd5c7c758972bf41aad954: Status 404 returned error can't find the container with id 46d9f8ef66f29e5af98c8e87328f117a5bc0a1af39dd5c7c758972bf41aad954
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.194773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.194826    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.194868    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-web-config\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.194897    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.194927    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfpl\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-kube-api-access-kvfpl\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.194970    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.195000    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.195033    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-out\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.195062    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.195087    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.195120    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.195157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.195397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.195179    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.216173 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.216142    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll"]
Jul 24 17:41:16.224937 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:16.224899    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73df4cb3_0d16_488a_a859_4d1a5de021d2.slice/crio-cacb969e353d0729fd57e01d07d0995e4eacce217ef6922c96c5a0a870d50f19 WatchSource:0}: Error finding container cacb969e353d0729fd57e01d07d0995e4eacce217ef6922c96c5a0a870d50f19: Status 404 returned error can't find the container with id cacb969e353d0729fd57e01d07d0995e4eacce217ef6922c96c5a0a870d50f19
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296022    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296082    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296120    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-web-config\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296150    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296180    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfpl\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-kube-api-access-kvfpl\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296220    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296253    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296288    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-out\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296315    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296389    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296440    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.296466    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.297422    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.298116 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:41:16.297568    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle podName:6ad7ffc9-7b3b-4800-81f8-a85d239b3527 nodeName:}" failed. No retries permitted until 2026-07-24 17:41:16.797547644 +0000 UTC m=+67.914474964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527") : configmap references non-existent config key: ca-bundle.crt
Jul 24 17:41:16.298947 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.298813    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.303292 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.303244    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.304493 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.304464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.305731 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.305687    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-volume\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.307584 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.307540    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfpl\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-kube-api-access-kvfpl\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.308432 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.308282    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.309592 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.308558    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-out\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.309592 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.308779    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.309592 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.309525    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.309890 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.309842    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-web-config\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.311391 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.311337    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.556649 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.556565    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll" event={"ID":"73df4cb3-0d16-488a-a859-4d1a5de021d2","Type":"ContainerStarted","Data":"cacb969e353d0729fd57e01d07d0995e4eacce217ef6922c96c5a0a870d50f19"}
Jul 24 17:41:16.558035 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.557967    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bwj9" event={"ID":"c2bbb19c-ed35-4308-a300-75b0e4021dd8","Type":"ContainerStarted","Data":"e8d8738c9b91351045582c85683452436c3bcbbd058a4573ecfad45fddcfcdbe"}
Jul 24 17:41:16.561528 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.561393    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj" event={"ID":"f02594fb-33a4-432f-84c4-1937c3077ccb","Type":"ContainerStarted","Data":"620965df6c2189acb5840baa01b66b3c0d671d1c18b34c5e38b3ac1e4ab1bbf0"}
Jul 24 17:41:16.561528 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.561449    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj" event={"ID":"f02594fb-33a4-432f-84c4-1937c3077ccb","Type":"ContainerStarted","Data":"d7561a81a0f59867bd90733e9f5b9801bd84f10143a16ec5045e5796bc296f02"}
Jul 24 17:41:16.561528 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.561494    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj" event={"ID":"f02594fb-33a4-432f-84c4-1937c3077ccb","Type":"ContainerStarted","Data":"46d9f8ef66f29e5af98c8e87328f117a5bc0a1af39dd5c7c758972bf41aad954"}
Jul 24 17:41:16.801875 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.800815    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:16.801875 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:16.801799    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:17.023232 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.023192    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74b54598bc-s4nqw"]
Jul 24 17:41:17.036910 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.036885    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:41:17.039094 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.038950    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.041517 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.041491    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 17:41:17.041889 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.041792    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74b54598bc-s4nqw"]
Jul 24 17:41:17.041889 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.041805    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 17:41:17.046210 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.045616    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a31bhr6tbkaub\""
Jul 24 17:41:17.046210 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.045887    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 17:41:17.046210 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.046086    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 17:41:17.046495 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.046465    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 17:41:17.046962 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.046747    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-z8csl\""
Jul 24 17:41:17.103246 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.103246 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103257    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96rl\" (UniqueName: \"kubernetes.io/projected/3adea86c-fe57-4909-b53c-b48e12f69fc6-kube-api-access-p96rl\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.103486 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103304    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adea86c-fe57-4909-b53c-b48e12f69fc6-metrics-client-ca\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.103486 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103392    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.103486 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103433    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.103486 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103463    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-tls\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.103739 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103502    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-grpc-tls\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.103739 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.103531    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205315    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205403    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205447    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-tls\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205492    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-grpc-tls\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205525    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205562    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205596    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96rl\" (UniqueName: \"kubernetes.io/projected/3adea86c-fe57-4909-b53c-b48e12f69fc6-kube-api-access-p96rl\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.205721 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.205644    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adea86c-fe57-4909-b53c-b48e12f69fc6-metrics-client-ca\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.208001 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.206449    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adea86c-fe57-4909-b53c-b48e12f69fc6-metrics-client-ca\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.213748 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.213719    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.215571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.215541    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.215680 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.215591    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.215941 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.215919    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-tls\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.216062 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.216048    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-grpc-tls\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.216295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.216278    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adea86c-fe57-4909-b53c-b48e12f69fc6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.226538 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.226496    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96rl\" (UniqueName: \"kubernetes.io/projected/3adea86c-fe57-4909-b53c-b48e12f69fc6-kube-api-access-p96rl\") pod \"thanos-querier-74b54598bc-s4nqw\" (UID: \"3adea86c-fe57-4909-b53c-b48e12f69fc6\") " pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:17.357826 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:17.357734    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:19.320184 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.320149    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86764cc475-q6smq"]
Jul 24 17:41:19.339918 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.339879    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86764cc475-q6smq"]
Jul 24 17:41:19.340086 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.340001    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.343496 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.343473    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 24 17:41:19.343635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.343473    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 24 17:41:19.343635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.343621    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 24 17:41:19.343761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.343696    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gzpcv\""
Jul 24 17:41:19.343761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.343744    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ch3h9b4v4hdg9\""
Jul 24 17:41:19.344005 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.343990    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 17:41:19.425973 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.425938    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-client-ca-bundle\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.426146 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.425983    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-secret-metrics-server-tls\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.426146 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.426050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5de00dc6-3126-46a4-969d-dff272cfaff2-metrics-server-audit-profiles\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.426238 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.426185    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5de00dc6-3126-46a4-969d-dff272cfaff2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.426238 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.426210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5de00dc6-3126-46a4-969d-dff272cfaff2-audit-log\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.426329 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.426237    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpcz\" (UniqueName: \"kubernetes.io/projected/5de00dc6-3126-46a4-969d-dff272cfaff2-kube-api-access-fzpcz\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.426329 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.426265    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-secret-metrics-server-client-certs\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.527665 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.527627    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5de00dc6-3126-46a4-969d-dff272cfaff2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.527848 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.527672    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5de00dc6-3126-46a4-969d-dff272cfaff2-audit-log\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.527848 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.527706    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpcz\" (UniqueName: \"kubernetes.io/projected/5de00dc6-3126-46a4-969d-dff272cfaff2-kube-api-access-fzpcz\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.527848 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.527733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-secret-metrics-server-client-certs\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.527848 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.527765    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-client-ca-bundle\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.527848 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.527803    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-secret-metrics-server-tls\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.527848 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.527828    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5de00dc6-3126-46a4-969d-dff272cfaff2-metrics-server-audit-profiles\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.528205 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.528183    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5de00dc6-3126-46a4-969d-dff272cfaff2-audit-log\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.528473 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.528437    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5de00dc6-3126-46a4-969d-dff272cfaff2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.528866 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.528844    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5de00dc6-3126-46a4-969d-dff272cfaff2-metrics-server-audit-profiles\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.530921 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.530897    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-secret-metrics-server-tls\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.531042 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.531021    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-secret-metrics-server-client-certs\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.531112 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.531080    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5de00dc6-3126-46a4-969d-dff272cfaff2-client-ca-bundle\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.535529 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.535505    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpcz\" (UniqueName: \"kubernetes.io/projected/5de00dc6-3126-46a4-969d-dff272cfaff2-kube-api-access-fzpcz\") pod \"metrics-server-86764cc475-q6smq\" (UID: \"5de00dc6-3126-46a4-969d-dff272cfaff2\") " pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.651508 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.651474    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:19.794457 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.794421    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"]
Jul 24 17:41:19.797680 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.797652    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:19.799932 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.799909    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 24 17:41:19.799932 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.799924    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xzfn2\""
Jul 24 17:41:19.806246 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.806226    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"]
Jul 24 17:41:19.932104 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.932007    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgbc\" (UniqueName: \"kubernetes.io/projected/24b312cc-b411-4e2d-ac8a-af2c28b0c83e-kube-api-access-nfgbc\") pod \"monitoring-plugin-68b7fc9574-v67b8\" (UID: \"24b312cc-b411-4e2d-ac8a-af2c28b0c83e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:19.932251 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:19.932151    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24b312cc-b411-4e2d-ac8a-af2c28b0c83e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-v67b8\" (UID: \"24b312cc-b411-4e2d-ac8a-af2c28b0c83e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:20.032992 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:20.032945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgbc\" (UniqueName: \"kubernetes.io/projected/24b312cc-b411-4e2d-ac8a-af2c28b0c83e-kube-api-access-nfgbc\") pod \"monitoring-plugin-68b7fc9574-v67b8\" (UID: \"24b312cc-b411-4e2d-ac8a-af2c28b0c83e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:20.033156 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:20.033044    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24b312cc-b411-4e2d-ac8a-af2c28b0c83e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-v67b8\" (UID: \"24b312cc-b411-4e2d-ac8a-af2c28b0c83e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:20.035997 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:20.035969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24b312cc-b411-4e2d-ac8a-af2c28b0c83e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-v67b8\" (UID: \"24b312cc-b411-4e2d-ac8a-af2c28b0c83e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:20.041921 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:20.041857    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgbc\" (UniqueName: \"kubernetes.io/projected/24b312cc-b411-4e2d-ac8a-af2c28b0c83e-kube-api-access-nfgbc\") pod \"monitoring-plugin-68b7fc9574-v67b8\" (UID: \"24b312cc-b411-4e2d-ac8a-af2c28b0c83e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:20.111901 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:20.111859    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:21.226734 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.224660    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:41:21.229121 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.229080    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.231997 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.231970    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 17:41:21.232139 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.232068    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 17:41:21.238225 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.238203    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 17:41:21.238356 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.238243    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 17:41:21.238356 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.238331    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ms2jm\""
Jul 24 17:41:21.238509 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.238427    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 17:41:21.238509 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.238210    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 17:41:21.238608 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.238550    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 17:41:21.238659 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.238627    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 17:41:21.239244 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.239212    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 17:41:21.239786 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.239516    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ave97vfkeefmo\""
Jul 24 17:41:21.239786 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.239560    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 17:41:21.239786 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.239572    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 17:41:21.240611 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.240023    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 17:41:21.240611 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.240226    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 17:41:21.240611 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.240442    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 17:41:21.256731 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.256699    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:41:21.344986 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.344921    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.344986 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.344979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345241 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345010    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9wl\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-kube-api-access-mg9wl\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345241 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345046    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-config\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345241 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345079    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345241 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345104    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345241 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345150    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345241 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345222    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345251    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345296    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345320    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345345    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345364    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345437    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345460    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.345504 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345499    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.346209 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345513    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.346209 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345531    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.346209 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345549    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.346209 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.345645    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446742    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446819    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9wl\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-kube-api-access-mg9wl\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446877    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-config\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446910    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446935    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.446975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446958    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.446984    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447006    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447052    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447077    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447112    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447138    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447171    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447194    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447238    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447264    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447288    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.447328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.447304    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.449126 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.448889    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.450309 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.450249    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.451559 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.451396    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.451736    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.451854    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.451906    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.452324    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.452488    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.452657    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.453474    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.453868    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.454406    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.455064    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.455558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.455473    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.456356 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.456332    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9wl\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-kube-api-access-mg9wl\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.457722 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.457233    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.457722 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.457292    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.457722 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.457600    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-config\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.458911 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.458868    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.459704 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.459659    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:21.530553 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.530469    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5c67v"
Jul 24 17:41:21.541441 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:21.541171    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:23.172546 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.172254    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86764cc475-q6smq"]
Jul 24 17:41:23.267924 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.266367    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:41:23.282735 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:41:23.277699    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e292caa_f56d_4eca_9233_88f23ca555d5.slice/crio-24f3eb08691f326151fdc7321e874d4ccde251fddb59cac215ab683d4e5b33bb WatchSource:0}: Error finding container 24f3eb08691f326151fdc7321e874d4ccde251fddb59cac215ab683d4e5b33bb: Status 404 returned error can't find the container with id 24f3eb08691f326151fdc7321e874d4ccde251fddb59cac215ab683d4e5b33bb
Jul 24 17:41:23.408489 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.408453    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74b54598bc-s4nqw"]
Jul 24 17:41:23.415316 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.415188    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"]
Jul 24 17:41:23.426399 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.426351    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:41:23.595623 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.595579    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll" event={"ID":"73df4cb3-0d16-488a-a859-4d1a5de021d2","Type":"ContainerStarted","Data":"cb1765c6b4e47311dba0e151acf0e345640e2da8ecb5f9285d09e99c8abc4a3f"}
Jul 24 17:41:23.595805 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.595661    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll" event={"ID":"73df4cb3-0d16-488a-a859-4d1a5de021d2","Type":"ContainerStarted","Data":"1f7e1b50e1d3b054ac783e7145db4a36072296447aa4e3f604f4bb9c3084e5a0"}
Jul 24 17:41:23.595805 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.595697    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll" event={"ID":"73df4cb3-0d16-488a-a859-4d1a5de021d2","Type":"ContainerStarted","Data":"476f237cccb732b64e24cfadba68b4ab4e5d9810b777463d0bb54ea427b674bb"}
Jul 24 17:41:23.597207 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.597179    2551 generic.go:356] "Generic (PLEG): container finished" podID="c2bbb19c-ed35-4308-a300-75b0e4021dd8" containerID="f126a667860a4a4ecfa9f2147f55c71d3b4a30450fa64ae82a89714968169701" exitCode=0
Jul 24 17:41:23.597334 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.597262    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bwj9" event={"ID":"c2bbb19c-ed35-4308-a300-75b0e4021dd8","Type":"ContainerDied","Data":"f126a667860a4a4ecfa9f2147f55c71d3b4a30450fa64ae82a89714968169701"}
Jul 24 17:41:23.600021 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.600003    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rc7j7" event={"ID":"00eca871-94cf-449c-aac1-ec4971d7a909","Type":"ContainerStarted","Data":"0f95cc2f14ca2b0953db943711f9aabe4004d81f2854e7b0aec968b224238abf"}
Jul 24 17:41:23.602411 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.602354    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj" event={"ID":"f02594fb-33a4-432f-84c4-1937c3077ccb","Type":"ContainerStarted","Data":"9113b3427970145ee5ae83795de6c957f1cec54256e57e04980738d829bb9ca2"}
Jul 24 17:41:23.603572 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.603547    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerStarted","Data":"7a41c77bee8a2788015b3b2dd66a6be77d87d44c248c6f2c04d35bfae4e47745"}
Jul 24 17:41:23.604989 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.604969    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-46rhf" event={"ID":"86dfdc0f-5604-4754-9a47-65b0d7c910fb","Type":"ContainerStarted","Data":"33e22cf19b5f78f3ec68267262635b1338131157a0bebed6363701166fd97ad8"}
Jul 24 17:41:23.605936 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.605916    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:23.607204 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.606820    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wnsct" event={"ID":"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb","Type":"ContainerStarted","Data":"62e349906491851ccac460b3311dbfd902dc02df79021132c74802c233b4e0b5"}
Jul 24 17:41:23.607204 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.606872    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wnsct" event={"ID":"ca6bd76d-63fe-4c44-a496-6f9ab029b2cb","Type":"ContainerStarted","Data":"356224d271911ef1af69f99e7311cb1421ae265a301e0cbe8ac0c4d073395c39"}
Jul 24 17:41:23.608356 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.608324    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f5gq5" event={"ID":"d346c91f-4715-48f3-873f-a9cb78c34f74","Type":"ContainerStarted","Data":"8d54825566908f663b7f6b9d2d64baa703dc9e10f472d463e2fd569a83a259b4"}
Jul 24 17:41:23.609747 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.609724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86764cc475-q6smq" event={"ID":"5de00dc6-3126-46a4-969d-dff272cfaff2","Type":"ContainerStarted","Data":"188c65e70d9e548fdb60f4caacff0cb0db736166bb5d1d02a50e4d6b990fcf49"}
Jul 24 17:41:23.610898 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.610877    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerStarted","Data":"24f3eb08691f326151fdc7321e874d4ccde251fddb59cac215ab683d4e5b33bb"}
Jul 24 17:41:23.611954 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.611936    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8" event={"ID":"24b312cc-b411-4e2d-ac8a-af2c28b0c83e","Type":"ContainerStarted","Data":"5a80ed9dcfb3d1817bf690601bfb2651f1219b6f77f6977f308bdc1a74010f7e"}
Jul 24 17:41:23.613129 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.613111    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" event={"ID":"3adea86c-fe57-4909-b53c-b48e12f69fc6","Type":"ContainerStarted","Data":"7571226fce6b52dd1c310fcb942eba50924dab2ddb75857b7c89a6651936e19c"}
Jul 24 17:41:23.616478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.616395    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nfmll" podStartSLOduration=1.954231288 podStartE2EDuration="8.616363635s" totalImagesPullingTime="6.662132347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:23.615410511 +0000 UTC m=+74.732337841" watchObservedRunningTime="2026-07-24 17:41:23.616363635 +0000 UTC m=+74.733290975"
Jul 24 17:41:23.632903 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.632858    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f5gq5" podStartSLOduration=66.89857048 podStartE2EDuration="1m14.632841687s" totalImagesPullingTime="7.734271207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:23.632205279 +0000 UTC m=+74.749132633" watchObservedRunningTime="2026-07-24 17:41:23.632841687 +0000 UTC m=+74.749769005"
Jul 24 17:41:23.654388 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.654315    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rc7j7" podStartSLOduration=2.175728585 podStartE2EDuration="19.65429454s" totalImagesPullingTime="17.478565955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:41:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:23.65386573 +0000 UTC m=+74.770793071" watchObservedRunningTime="2026-07-24 17:41:23.65429454 +0000 UTC m=+74.771221881"
Jul 24 17:41:23.677549 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.675390    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wnsct" podStartSLOduration=67.290102708 podStartE2EDuration="1m14.675352107s" totalImagesPullingTime="7.385249399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:23.67336915 +0000 UTC m=+74.790296491" watchObservedRunningTime="2026-07-24 17:41:23.675352107 +0000 UTC m=+74.792279451"
Jul 24 17:41:23.692012 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.691945    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-46rhf" podStartSLOduration=67.293353209 podStartE2EDuration="1m14.691926263s" totalImagesPullingTime="7.398573054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:23.690683048 +0000 UTC m=+74.807610391" watchObservedRunningTime="2026-07-24 17:41:23.691926263 +0000 UTC m=+74.808853604"
Jul 24 17:41:23.726549 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:23.726503    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pxsnj" podStartSLOduration=3.326224516 podStartE2EDuration="9.726486858s" totalImagesPullingTime="6.400262342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:23.725426883 +0000 UTC m=+74.842354224" watchObservedRunningTime="2026-07-24 17:41:23.726486858 +0000 UTC m=+74.843414200"
Jul 24 17:41:24.624126 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:24.623886    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bwj9" event={"ID":"c2bbb19c-ed35-4308-a300-75b0e4021dd8","Type":"ContainerStarted","Data":"9e8afb9d0cef608e1640a7cfff9671ca8a23dae76ca2d01f95e81f141721c469"}
Jul 24 17:41:24.624126 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:24.623977    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4bwj9" event={"ID":"c2bbb19c-ed35-4308-a300-75b0e4021dd8","Type":"ContainerStarted","Data":"69003fb3786de625298aab90ffc5fec94f36c2d6cb59558ba3a2ada3914aacbb"}
Jul 24 17:41:26.636408 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.636299    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86764cc475-q6smq" event={"ID":"5de00dc6-3126-46a4-969d-dff272cfaff2","Type":"ContainerStarted","Data":"cb146faea04e60391d738b8e825abf407e5af4a3e396f3203d9b4f5a8ba56525"}
Jul 24 17:41:26.637953 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.637927    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c" exitCode=0
Jul 24 17:41:26.638073 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.637991    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"}
Jul 24 17:41:26.639801 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.639761    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8" event={"ID":"24b312cc-b411-4e2d-ac8a-af2c28b0c83e","Type":"ContainerStarted","Data":"3dbef75e3469650f3672739c6eedcfe7eaf458386fb99755da59cc2ab8133928"}
Jul 24 17:41:26.639974 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.639956    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:26.642310 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.642284    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" event={"ID":"3adea86c-fe57-4909-b53c-b48e12f69fc6","Type":"ContainerStarted","Data":"148e6252ffbfbf3fd9ab2e2134bdf9220f1c436025fb963f57779360987dc949"}
Jul 24 17:41:26.642428 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.642352    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" event={"ID":"3adea86c-fe57-4909-b53c-b48e12f69fc6","Type":"ContainerStarted","Data":"936bbda86c9364dfa87e7cd0d9b86dc51a74fe98eaf3cfa0da508ab07c2fdb5f"}
Jul 24 17:41:26.642428 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.642423    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" event={"ID":"3adea86c-fe57-4909-b53c-b48e12f69fc6","Type":"ContainerStarted","Data":"3777fe842f4b3219dd0a4e008a75692fa9c5146dc075214ee0a5bde8d579a267"}
Jul 24 17:41:26.643956 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.643935    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerID="432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718" exitCode=0
Jul 24 17:41:26.644061 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.643966    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718"}
Jul 24 17:41:26.645295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.645276    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8"
Jul 24 17:41:26.654933 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.654892    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4bwj9" podStartSLOduration=4.577392376 podStartE2EDuration="11.654874332s" totalImagesPullingTime="6.892062317s" totalInitContainerRuntime="185.419639ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:24.643956463 +0000 UTC m=+75.760883804" watchObservedRunningTime="2026-07-24 17:41:26.654874332 +0000 UTC m=+77.771801687"
Jul 24 17:41:26.659605 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.655779    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86764cc475-q6smq" podStartSLOduration=4.793455247 podStartE2EDuration="7.655767967s" totalImagesPullingTime="2.86231272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:26.654600496 +0000 UTC m=+77.771527836" watchObservedRunningTime="2026-07-24 17:41:26.655767967 +0000 UTC m=+77.772695311"
Jul 24 17:41:26.673581 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:26.673537    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-v67b8" podStartSLOduration=5.072260098 podStartE2EDuration="7.673519605s" totalImagesPullingTime="2.601259507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:26.672601246 +0000 UTC m=+77.789528587" watchObservedRunningTime="2026-07-24 17:41:26.673519605 +0000 UTC m=+77.790446948"
Jul 24 17:41:27.654839 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:27.654199    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" event={"ID":"3adea86c-fe57-4909-b53c-b48e12f69fc6","Type":"ContainerStarted","Data":"a46d3ccdaaf6d7ffdab778e3a7d465dd3f39484df983356da457de98d0e9f53d"}
Jul 24 17:41:27.654839 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:27.654286    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" event={"ID":"3adea86c-fe57-4909-b53c-b48e12f69fc6","Type":"ContainerStarted","Data":"d90b53f6104f2a06a826372f5ac8260771cff44054bd7e01da2e64b5296de7b5"}
Jul 24 17:41:27.654839 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:27.654332    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" event={"ID":"3adea86c-fe57-4909-b53c-b48e12f69fc6","Type":"ContainerStarted","Data":"9a210bdffbd8e80b0f4c93d412d5fd9446d382d7e41d0728984d5ef3d11be875"}
Jul 24 17:41:27.680767 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:27.680716    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw" podStartSLOduration=7.347524637 podStartE2EDuration="10.680695341s" totalImagesPullingTime="3.333170704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:27.678597944 +0000 UTC m=+78.795525285" watchObservedRunningTime="2026-07-24 17:41:27.680695341 +0000 UTC m=+78.797622681"
Jul 24 17:41:28.663832 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.663157    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerStarted","Data":"0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160"}
Jul 24 17:41:28.663832 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.663276    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerStarted","Data":"36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be"}
Jul 24 17:41:28.663832 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.663320    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerStarted","Data":"3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f"}
Jul 24 17:41:28.664391 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.663905    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:28.664391 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.663978    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerStarted","Data":"05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be"}
Jul 24 17:41:28.664391 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.664027    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerStarted","Data":"82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882"}
Jul 24 17:41:28.664391 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.664068    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerStarted","Data":"6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3"}
Jul 24 17:41:28.693444 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:28.693370    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=8.585275673 podStartE2EDuration="12.693353069s" totalImagesPullingTime="4.075934687s" totalInitContainerRuntime="32.142709ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:28.690913105 +0000 UTC m=+79.807840446" watchObservedRunningTime="2026-07-24 17:41:28.693353069 +0000 UTC m=+79.810280409"
Jul 24 17:41:30.675062 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:30.675025    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerStarted","Data":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"}
Jul 24 17:41:30.675478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:30.675090    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerStarted","Data":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"}
Jul 24 17:41:30.675478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:30.675116    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerStarted","Data":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"}
Jul 24 17:41:30.675478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:30.675143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerStarted","Data":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"}
Jul 24 17:41:30.675478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:30.675166    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerStarted","Data":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"}
Jul 24 17:41:30.675478 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:30.675188    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerStarted","Data":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"}
Jul 24 17:41:30.703267 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:30.703195    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.586765415 podStartE2EDuration="9.703146397s" totalImagesPullingTime="6.092126699s" totalInitContainerRuntime="24.254283ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:41:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:41:30.701811084 +0000 UTC m=+81.818738423" watchObservedRunningTime="2026-07-24 17:41:30.703146397 +0000 UTC m=+81.820073782"
Jul 24 17:41:31.542617 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:31.542584    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:41:34.673244 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:34.673217    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74b54598bc-s4nqw"
Jul 24 17:41:39.652663 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:39.652631    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:39.653066 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:39.652691    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:54.628753 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:54.628723    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-46rhf"
Jul 24 17:41:59.657882 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:59.657853    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:41:59.661614 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:41:59.661588    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86764cc475-q6smq"
Jul 24 17:42:21.542583 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:21.542546    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:21.561396 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:21.561355    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:21.898573 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:21.898544    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:35.313702 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:35.313669    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:42:35.948480 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:35.948433    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="alertmanager" containerID="cri-o://6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3" gracePeriod=120
Jul 24 17:42:35.948686 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:35.948533    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="prom-label-proxy" containerID="cri-o://0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160" gracePeriod=120
Jul 24 17:42:35.948686 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:35.948566    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="config-reloader" containerID="cri-o://82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882" gracePeriod=120
Jul 24 17:42:35.948686 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:35.948566    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy" containerID="cri-o://3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f" gracePeriod=120
Jul 24 17:42:35.948686 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:35.948511    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-metric" containerID="cri-o://36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be" gracePeriod=120
Jul 24 17:42:35.948686 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:35.948523    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-web" containerID="cri-o://05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be" gracePeriod=120
Jul 24 17:42:36.955104 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955072    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerID="0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160" exitCode=0
Jul 24 17:42:36.955104 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955097    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerID="36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be" exitCode=0
Jul 24 17:42:36.955104 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955105    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerID="3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f" exitCode=0
Jul 24 17:42:36.955104 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955111    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerID="82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882" exitCode=0
Jul 24 17:42:36.955104 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955116    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerID="6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3" exitCode=0
Jul 24 17:42:36.955591 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955136    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160"}
Jul 24 17:42:36.955591 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955197    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be"}
Jul 24 17:42:36.955591 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955224    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f"}
Jul 24 17:42:36.955591 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955250    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882"}
Jul 24 17:42:36.955591 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:36.955273    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3"}
Jul 24 17:42:37.185785 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.185761    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:37.244900 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.244807    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-main-tls\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.244900 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.244862    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-out\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.244900 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.244895    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-web\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245171 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.244937    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-main-db\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245171 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.244962    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-volume\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245171 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.244984    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245171 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245007    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-tls-assets\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245171 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245032    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245171 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245106    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-web-config\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245171 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245132    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvfpl\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-kube-api-access-kvfpl\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245535 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245180    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-cluster-tls-config\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245535 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245209    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-metrics-client-ca\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245535 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245269    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\" (UID: \"6ad7ffc9-7b3b-4800-81f8-a85d239b3527\") "
Jul 24 17:42:37.245535 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245350    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:42:37.245746 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.245586    2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-main-db\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.246470 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.246396    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:37.246781 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.246755    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:37.248357 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.248310    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:37.248801 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.248762    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:37.249008 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.248933    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:42:37.249297 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.249061    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-volume" (OuterVolumeSpecName: "config-volume") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:37.249297 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.249250    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:37.249911 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.249890    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-kube-api-access-kvfpl" (OuterVolumeSpecName: "kube-api-access-kvfpl") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "kube-api-access-kvfpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:42:37.250194 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.250156    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-out" (OuterVolumeSpecName: "config-out") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:42:37.250580 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.250555    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:37.254185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.254158    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:37.262651 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.262615    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-web-config" (OuterVolumeSpecName: "web-config") pod "6ad7ffc9-7b3b-4800-81f8-a85d239b3527" (UID: "6ad7ffc9-7b3b-4800-81f8-a85d239b3527"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:37.346739 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346707    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346739 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346735    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-main-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346751    2551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-out\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346763    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346775    2551 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346787    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346799    2551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-tls-assets\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346811    2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346822    2551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-web-config\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346833    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfpl\" (UniqueName: \"kubernetes.io/projected/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-kube-api-access-kvfpl\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346844    2551 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-cluster-tls-config\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.346950 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.346855    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad7ffc9-7b3b-4800-81f8-a85d239b3527-metrics-client-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:37.961985 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.961955    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerID="05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be" exitCode=0
Jul 24 17:42:37.962423 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.962045    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be"}
Jul 24 17:42:37.962423 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.962110    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6ad7ffc9-7b3b-4800-81f8-a85d239b3527","Type":"ContainerDied","Data":"7a41c77bee8a2788015b3b2dd66a6be77d87d44c248c6f2c04d35bfae4e47745"}
Jul 24 17:42:37.962423 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.962055    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:37.962423 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.962153    2551 scope.go:117] "RemoveContainer" containerID="0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160"
Jul 24 17:42:37.970699 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.970684    2551 scope.go:117] "RemoveContainer" containerID="36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be"
Jul 24 17:42:37.977674 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.977658    2551 scope.go:117] "RemoveContainer" containerID="3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f"
Jul 24 17:42:37.984645 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.984572    2551 scope.go:117] "RemoveContainer" containerID="05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be"
Jul 24 17:42:37.987023 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.986998    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:42:37.991328 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.991300    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:42:37.993539 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:37.993526    2551 scope.go:117] "RemoveContainer" containerID="82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882"
Jul 24 17:42:38.000657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.000642    2551 scope.go:117] "RemoveContainer" containerID="6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3"
Jul 24 17:42:38.007672 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.007652    2551 scope.go:117] "RemoveContainer" containerID="432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718"
Jul 24 17:42:38.014802 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.014786    2551 scope.go:117] "RemoveContainer" containerID="0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160"
Jul 24 17:42:38.015102 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:38.015081    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160\": container with ID starting with 0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160 not found: ID does not exist" containerID="0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160"
Jul 24 17:42:38.015148 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015114    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160"} err="failed to get container status \"0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160\": rpc error: code = NotFound desc = could not find container \"0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160\": container with ID starting with 0ab0397baff892823e1fbb7210bcb781fe0f678c820afe22cdae5fa89227c160 not found: ID does not exist"
Jul 24 17:42:38.015188 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015149    2551 scope.go:117] "RemoveContainer" containerID="36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be"
Jul 24 17:42:38.015393 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:38.015362    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be\": container with ID starting with 36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be not found: ID does not exist" containerID="36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be"
Jul 24 17:42:38.015440 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015402    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be"} err="failed to get container status \"36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be\": rpc error: code = NotFound desc = could not find container \"36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be\": container with ID starting with 36dbb6014de9332b659d0802d282977991aa6bd07dc851a7910abc74649857be not found: ID does not exist"
Jul 24 17:42:38.015440 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015419    2551 scope.go:117] "RemoveContainer" containerID="3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f"
Jul 24 17:42:38.015634 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:38.015615    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f\": container with ID starting with 3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f not found: ID does not exist" containerID="3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f"
Jul 24 17:42:38.015694 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015639    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f"} err="failed to get container status \"3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f\": rpc error: code = NotFound desc = could not find container \"3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f\": container with ID starting with 3e2b40dfba5e7ec173bd012e7f4044a9595a7c787863cf5d80ed412a5c677f8f not found: ID does not exist"
Jul 24 17:42:38.015694 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015652    2551 scope.go:117] "RemoveContainer" containerID="05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be"
Jul 24 17:42:38.015865 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:38.015841    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be\": container with ID starting with 05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be not found: ID does not exist" containerID="05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be"
Jul 24 17:42:38.015901 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015869    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be"} err="failed to get container status \"05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be\": rpc error: code = NotFound desc = could not find container \"05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be\": container with ID starting with 05fe7d55e4d284e7fe41270dfd8a5c1448f8068e78c089907f145435e017a8be not found: ID does not exist"
Jul 24 17:42:38.015901 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.015882    2551 scope.go:117] "RemoveContainer" containerID="82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882"
Jul 24 17:42:38.016118 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:38.016102    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882\": container with ID starting with 82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882 not found: ID does not exist" containerID="82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882"
Jul 24 17:42:38.016185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.016125    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882"} err="failed to get container status \"82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882\": rpc error: code = NotFound desc = could not find container \"82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882\": container with ID starting with 82ee456b1a0024c15be44016d88fe6195055a2f3917896eb4a9ee5154a462882 not found: ID does not exist"
Jul 24 17:42:38.016185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.016139    2551 scope.go:117] "RemoveContainer" containerID="6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3"
Jul 24 17:42:38.016387 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:38.016359    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3\": container with ID starting with 6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3 not found: ID does not exist" containerID="6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3"
Jul 24 17:42:38.016436 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.016393    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3"} err="failed to get container status \"6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3\": rpc error: code = NotFound desc = could not find container \"6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3\": container with ID starting with 6a59031a1f1d56879861a108e57c0ec10f230fa5749a7ffe9086bb6d379a43b3 not found: ID does not exist"
Jul 24 17:42:38.016436 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.016407    2551 scope.go:117] "RemoveContainer" containerID="432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718"
Jul 24 17:42:38.016628 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:38.016611    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718\": container with ID starting with 432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718 not found: ID does not exist" containerID="432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718"
Jul 24 17:42:38.016722 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.016628    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718"} err="failed to get container status \"432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718\": rpc error: code = NotFound desc = could not find container \"432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718\": container with ID starting with 432835cd777d807c2c25cdf71df467f2f7769b0da6540ee58600d6f4232e7718 not found: ID does not exist"
Jul 24 17:42:38.023784 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.023764    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:42:38.024055 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024043    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="init-config-reloader"
Jul 24 17:42:38.024098 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024057    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="init-config-reloader"
Jul 24 17:42:38.024098 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024069    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-metric"
Jul 24 17:42:38.024098 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024075    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-metric"
Jul 24 17:42:38.024098 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024085    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="config-reloader"
Jul 24 17:42:38.024098 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024092    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="config-reloader"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024103    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="alertmanager"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024108    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="alertmanager"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024114    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="prom-label-proxy"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024118    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="prom-label-proxy"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024125    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-web"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024131    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-web"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024138    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024143    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024185    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-web"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024194    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="config-reloader"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024203    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="alertmanager"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024208    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="prom-label-proxy"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024215    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy-metric"
Jul 24 17:42:38.024242 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.024220    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" containerName="kube-rbac-proxy"
Jul 24 17:42:38.029027 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.029009    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.031644 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031610    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 17:42:38.031644 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031611    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 17:42:38.031814 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031612    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 17:42:38.031814 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031664    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 17:42:38.031814 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031630    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 17:42:38.031814 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031612    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 17:42:38.031814 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031629    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pq2mq\""
Jul 24 17:42:38.032178 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.031943    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 17:42:38.032178 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.032047    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 17:42:38.036898 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.036880    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 17:42:38.043612 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.043591    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:42:38.052416 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052393    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052522 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052445    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052522 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052478    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64893af6-084d-431a-b0d2-846b467493f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052522 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052512    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64893af6-084d-431a-b0d2-846b467493f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052664 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052544    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64893af6-084d-431a-b0d2-846b467493f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052664 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052571    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64893af6-084d-431a-b0d2-846b467493f4-config-out\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052664 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64893af6-084d-431a-b0d2-846b467493f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052664 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052639    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbzv\" (UniqueName: \"kubernetes.io/projected/64893af6-084d-431a-b0d2-846b467493f4-kube-api-access-nlbzv\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052869 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052678    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052869 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052722    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052869 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052748    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052869 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052779    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-web-config\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.052869 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.052844    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154197 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154166    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64893af6-084d-431a-b0d2-846b467493f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154197 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154199    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64893af6-084d-431a-b0d2-846b467493f4-config-out\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154489 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154218    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64893af6-084d-431a-b0d2-846b467493f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154489 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbzv\" (UniqueName: \"kubernetes.io/projected/64893af6-084d-431a-b0d2-846b467493f4-kube-api-access-nlbzv\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154489 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154254    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154489 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154281    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154489 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154489 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154472    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-web-config\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154763 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154535    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154763 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154763 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154617    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154763 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154648    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64893af6-084d-431a-b0d2-846b467493f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.154763 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.154687    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64893af6-084d-431a-b0d2-846b467493f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.155183 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.155010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/64893af6-084d-431a-b0d2-846b467493f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.155312 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.155290    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64893af6-084d-431a-b0d2-846b467493f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.156861 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.156814    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64893af6-084d-431a-b0d2-846b467493f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.157999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.157451    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.157999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.157907    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-web-config\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.157999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.157941    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.158255 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.158100    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64893af6-084d-431a-b0d2-846b467493f4-config-out\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.158255 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.158246    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64893af6-084d-431a-b0d2-846b467493f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.158476 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.158455    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.158529 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.158504    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.158887 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.158872    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.159280 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.159264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64893af6-084d-431a-b0d2-846b467493f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.164404 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.164364    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbzv\" (UniqueName: \"kubernetes.io/projected/64893af6-084d-431a-b0d2-846b467493f4-kube-api-access-nlbzv\") pod \"alertmanager-main-0\" (UID: \"64893af6-084d-431a-b0d2-846b467493f4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.338980 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.338949    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 17:42:38.481674 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.481649    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 17:42:38.483724 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:42:38.483701    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64893af6_084d_431a_b0d2_846b467493f4.slice/crio-df36aaa98eaa62d6bbe3f0912080840854a80394cd33be12fb77e4011e894f51 WatchSource:0}: Error finding container df36aaa98eaa62d6bbe3f0912080840854a80394cd33be12fb77e4011e894f51: Status 404 returned error can't find the container with id df36aaa98eaa62d6bbe3f0912080840854a80394cd33be12fb77e4011e894f51
Jul 24 17:42:38.967915 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.967828    2551 generic.go:356] "Generic (PLEG): container finished" podID="64893af6-084d-431a-b0d2-846b467493f4" containerID="09d940b7e205131df44820f670d44558999137fc6b8704ad76cb501caff835e4" exitCode=0
Jul 24 17:42:38.968279 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.967918    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerDied","Data":"09d940b7e205131df44820f670d44558999137fc6b8704ad76cb501caff835e4"}
Jul 24 17:42:38.968279 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:38.968005    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerStarted","Data":"df36aaa98eaa62d6bbe3f0912080840854a80394cd33be12fb77e4011e894f51"}
Jul 24 17:42:39.305331 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.305301    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad7ffc9-7b3b-4800-81f8-a85d239b3527" path="/var/lib/kubelet/pods/6ad7ffc9-7b3b-4800-81f8-a85d239b3527/volumes"
Jul 24 17:42:39.326294 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.326261    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dcc569685-tk72l"]
Jul 24 17:42:39.328527 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.328511    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.331617 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.331585    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 24 17:42:39.332865 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.332827    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 24 17:42:39.333100 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.333080    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 24 17:42:39.333208 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.333181    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2sqrj\""
Jul 24 17:42:39.333273 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.333187    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 24 17:42:39.333520 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.333502    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 24 17:42:39.342931 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.342911    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 24 17:42:39.352929 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.352900    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dcc569685-tk72l"]
Jul 24 17:42:39.364971 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.364946    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-federate-client-tls\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.365107 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.364996    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pk45\" (UniqueName: \"kubernetes.io/projected/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-kube-api-access-2pk45\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.365107 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.365081    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.365239 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.365128    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-telemeter-client-tls\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.365239 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.365159    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.365239 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.365220    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-secret-telemeter-client\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.365407 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.365297    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-serving-certs-ca-bundle\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.365407 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.365324    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-metrics-client-ca\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.466767 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.466722    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pk45\" (UniqueName: \"kubernetes.io/projected/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-kube-api-access-2pk45\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.466962 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.466794    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.466962 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.466850    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-telemeter-client-tls\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.466962 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.466875    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.466962 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.466916    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-secret-telemeter-client\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.466962 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.466961    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-serving-certs-ca-bundle\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.467195 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.466984    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-metrics-client-ca\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.467195 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.467045    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-federate-client-tls\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.468089 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.468059    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-serving-certs-ca-bundle\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.468406 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.468059    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-metrics-client-ca\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.468564 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.468535    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.470051 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.469997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-secret-telemeter-client\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.470136 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.470093    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.470178 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.470148    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-telemeter-client-tls\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.470280 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.470262    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-federate-client-tls\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.477890 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.477867    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pk45\" (UniqueName: \"kubernetes.io/projected/4403fc70-3600-4f5f-bd5d-3320f9fa9f7a-kube-api-access-2pk45\") pod \"telemeter-client-5dcc569685-tk72l\" (UID: \"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a\") " pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.641482 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.641442    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l"
Jul 24 17:42:39.645066 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.645036    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:42:39.645403 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.645355    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="prometheus" containerID="cri-o://321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97" gracePeriod=600
Jul 24 17:42:39.645495 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.645401    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-web" containerID="cri-o://53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64" gracePeriod=600
Jul 24 17:42:39.645546 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.645477    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="config-reloader" containerID="cri-o://3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f" gracePeriod=600
Jul 24 17:42:39.645609 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.645481    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="thanos-sidecar" containerID="cri-o://b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe" gracePeriod=600
Jul 24 17:42:39.645609 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.645564    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy" containerID="cri-o://45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65" gracePeriod=600
Jul 24 17:42:39.645773 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.645364    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648" gracePeriod=600
Jul 24 17:42:39.812418 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.812368    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dcc569685-tk72l"]
Jul 24 17:42:39.814007 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:42:39.813980    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4403fc70_3600_4f5f_bd5d_3320f9fa9f7a.slice/crio-41341fffe2a836f9e9bcd9ffc3a71f5b43cdcd4a56df8d2c27bafefdea1ce60d WatchSource:0}: Error finding container 41341fffe2a836f9e9bcd9ffc3a71f5b43cdcd4a56df8d2c27bafefdea1ce60d: Status 404 returned error can't find the container with id 41341fffe2a836f9e9bcd9ffc3a71f5b43cdcd4a56df8d2c27bafefdea1ce60d
Jul 24 17:42:39.930641 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.930616    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971360    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-kubelet-serving-ca-bundle\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971441    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-grpc-tls\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971473    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-2\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971506    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-0\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971552    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-config-out\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971577    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-metrics-client-ca\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971608    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971635    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-1\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971674    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-trusted-ca-bundle\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971716    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-web-config\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971744    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-metrics-client-certs\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971769    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-thanos-prometheus-http-client-file\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971811    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg9wl\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-kube-api-access-mg9wl\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971841    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-db\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971883    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-tls\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971906    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-tls-assets\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971937    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-serving-certs-ca-bundle\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.972964 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.971978    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-config\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.974421 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.972008    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.974421 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.972051    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-kube-rbac-proxy\") pod \"2e292caa-f56d-4eca-9233-88f23ca555d5\" (UID: \"2e292caa-f56d-4eca-9233-88f23ca555d5\") "
Jul 24 17:42:39.974421 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.973085    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:39.974421 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.973537    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:39.974638 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.974578    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:39.978186 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.974907    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:39.978186 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.975195    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.978186 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.975248    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:39.978186 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.977921    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:42:39.978515 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.978304    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:39.978515 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.978450    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.979099 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.979072    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.979518 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.979495    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 17:42:39.980446 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.980287    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:42:39.980446 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.980368    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-config-out" (OuterVolumeSpecName: "config-out") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:42:39.980658 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.980615    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.982050 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.982012    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-kube-api-access-mg9wl" (OuterVolumeSpecName: "kube-api-access-mg9wl") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "kube-api-access-mg9wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:42:39.983514 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.983477    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.983606 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.983562    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.984395 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984343    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-config" (OuterVolumeSpecName: "config") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.984672 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984643    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648" exitCode=0
Jul 24 17:42:39.984740 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984676    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65" exitCode=0
Jul 24 17:42:39.984740 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984687    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64" exitCode=0
Jul 24 17:42:39.984740 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984696    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe" exitCode=0
Jul 24 17:42:39.984740 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984705    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f" exitCode=0
Jul 24 17:42:39.984740 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984713    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97" exitCode=0
Jul 24 17:42:39.984960 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984792    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"}
Jul 24 17:42:39.984960 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984864    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"}
Jul 24 17:42:39.984960 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984910    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"}
Jul 24 17:42:39.984960 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984949    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"}
Jul 24 17:42:39.985139 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.984988    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"}
Jul 24 17:42:39.985139 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.985030    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"}
Jul 24 17:42:39.985139 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.985069    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2e292caa-f56d-4eca-9233-88f23ca555d5","Type":"ContainerDied","Data":"24f3eb08691f326151fdc7321e874d4ccde251fddb59cac215ab683d4e5b33bb"}
Jul 24 17:42:39.985139 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.985113    2551 scope.go:117] "RemoveContainer" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:39.985635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.985286    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:39.987795 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.987644    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:39.988943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.988851    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l" event={"ID":"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a","Type":"ContainerStarted","Data":"41341fffe2a836f9e9bcd9ffc3a71f5b43cdcd4a56df8d2c27bafefdea1ce60d"}
Jul 24 17:42:39.999163 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:39.999113    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-web-config" (OuterVolumeSpecName: "web-config") pod "2e292caa-f56d-4eca-9233-88f23ca555d5" (UID: "2e292caa-f56d-4eca-9233-88f23ca555d5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:42:40.002544 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.002510    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerStarted","Data":"6059015c7e1f8abe2c0df3b772456948d3735e1e26059ca1ec1d54d65de2425d"}
Jul 24 17:42:40.002678 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.002620    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerStarted","Data":"af000e00a00e7d2207288229e6277ac199d880ead9c324ae6760df6022b7c52e"}
Jul 24 17:42:40.002678 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.002660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerStarted","Data":"28bb11775d26d2ba7cec0155e7681a3cff505338bbd5f0cdc1b403ad9d4b33e5"}
Jul 24 17:42:40.002781 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.002696    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerStarted","Data":"b28635bbbaf38f3379d53be29bae6c4024fc939fb723adf3ef951f50df199e90"}
Jul 24 17:42:40.002781 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.002731    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerStarted","Data":"b1584d4c09c60417de5fe64d2b7147e018ec1591d16f97c00db7e3d24eebf44e"}
Jul 24 17:42:40.002781 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.002773    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"64893af6-084d-431a-b0d2-846b467493f4","Type":"ContainerStarted","Data":"7b245b0932d6fce04329e9328d9c6cadabaa7e8ae79aa5e387e2aed917275fe9"}
Jul 24 17:42:40.003930 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.003914    2551 scope.go:117] "RemoveContainer" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.012348 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.012320    2551 scope.go:117] "RemoveContainer" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.035440 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.035335    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.023315809 podStartE2EDuration="2.035319123s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.003314ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:40.033547881 +0000 UTC m=+151.150475252" watchObservedRunningTime="2026-07-24 17:42:40.035319123 +0000 UTC m=+151.152246463"
Jul 24 17:42:40.049446 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.049411    2551 scope.go:117] "RemoveContainer" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.057505 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.057483    2551 scope.go:117] "RemoveContainer" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.065252 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.065227    2551 scope.go:117] "RemoveContainer" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073293    2551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-config-out\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073316    2551 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-metrics-client-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073332    2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073349    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073363    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073396    2551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-web-config\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073410    2551 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-metrics-client-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073422    2551 scope.go:117] "RemoveContainer" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073425    2551 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073523    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg9wl\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-kube-api-access-mg9wl\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073537    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-db\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073546    2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073569    2551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e292caa-f56d-4eca-9233-88f23ca555d5-tls-assets\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073588    2551 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073602    2551 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-config\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073616    2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073633    2551 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-kube-rbac-proxy\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073646    2551 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.073657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073660    2551 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e292caa-f56d-4eca-9233-88f23ca555d5-secret-grpc-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.074323 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073673    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.074323 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.073687    2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e292caa-f56d-4eca-9233-88f23ca555d5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:42:40.081129 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.081111    2551 scope.go:117] "RemoveContainer" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:40.081448 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:40.081423    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": container with ID starting with 6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648 not found: ID does not exist" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:40.081521 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.081459    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"} err="failed to get container status \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": rpc error: code = NotFound desc = could not find container \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": container with ID starting with 6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648 not found: ID does not exist"
Jul 24 17:42:40.081521 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.081485    2551 scope.go:117] "RemoveContainer" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.081774 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:40.081755    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": container with ID starting with 45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65 not found: ID does not exist" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.081820 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.081780    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"} err="failed to get container status \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": rpc error: code = NotFound desc = could not find container \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": container with ID starting with 45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65 not found: ID does not exist"
Jul 24 17:42:40.081820 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.081799    2551 scope.go:117] "RemoveContainer" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.082052 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:40.082038    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": container with ID starting with 53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64 not found: ID does not exist" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.082090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082054    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"} err="failed to get container status \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": rpc error: code = NotFound desc = could not find container \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": container with ID starting with 53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64 not found: ID does not exist"
Jul 24 17:42:40.082090 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082069    2551 scope.go:117] "RemoveContainer" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.082283 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:40.082270    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": container with ID starting with b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe not found: ID does not exist" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.082325 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082287    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"} err="failed to get container status \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": rpc error: code = NotFound desc = could not find container \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": container with ID starting with b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe not found: ID does not exist"
Jul 24 17:42:40.082325 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082299    2551 scope.go:117] "RemoveContainer" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.082546 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:40.082528    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": container with ID starting with 3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f not found: ID does not exist" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.082590 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082553    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"} err="failed to get container status \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": rpc error: code = NotFound desc = could not find container \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": container with ID starting with 3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f not found: ID does not exist"
Jul 24 17:42:40.082590 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082570    2551 scope.go:117] "RemoveContainer" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.082864 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:40.082841    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": container with ID starting with 321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97 not found: ID does not exist" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.082917 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082869    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"} err="failed to get container status \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": rpc error: code = NotFound desc = could not find container \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": container with ID starting with 321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97 not found: ID does not exist"
Jul 24 17:42:40.082917 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.082885    2551 scope.go:117] "RemoveContainer" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.083135 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:42:40.083117    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": container with ID starting with 559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c not found: ID does not exist" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.083193 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083138    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"} err="failed to get container status \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": rpc error: code = NotFound desc = could not find container \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": container with ID starting with 559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c not found: ID does not exist"
Jul 24 17:42:40.083193 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083151    2551 scope.go:117] "RemoveContainer" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:40.083409 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083363    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"} err="failed to get container status \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": rpc error: code = NotFound desc = could not find container \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": container with ID starting with 6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648 not found: ID does not exist"
Jul 24 17:42:40.083467 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083412    2551 scope.go:117] "RemoveContainer" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.083691 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083669    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"} err="failed to get container status \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": rpc error: code = NotFound desc = could not find container \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": container with ID starting with 45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65 not found: ID does not exist"
Jul 24 17:42:40.083758 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083693    2551 scope.go:117] "RemoveContainer" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.083985 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083966    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"} err="failed to get container status \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": rpc error: code = NotFound desc = could not find container \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": container with ID starting with 53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64 not found: ID does not exist"
Jul 24 17:42:40.084031 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.083986    2551 scope.go:117] "RemoveContainer" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.084220 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084194    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"} err="failed to get container status \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": rpc error: code = NotFound desc = could not find container \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": container with ID starting with b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe not found: ID does not exist"
Jul 24 17:42:40.084259 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084223    2551 scope.go:117] "RemoveContainer" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.084522 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084494    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"} err="failed to get container status \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": rpc error: code = NotFound desc = could not find container \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": container with ID starting with 3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f not found: ID does not exist"
Jul 24 17:42:40.084522 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084521    2551 scope.go:117] "RemoveContainer" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.084746 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084729    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"} err="failed to get container status \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": rpc error: code = NotFound desc = could not find container \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": container with ID starting with 321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97 not found: ID does not exist"
Jul 24 17:42:40.084799 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084749    2551 scope.go:117] "RemoveContainer" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.084968 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084948    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"} err="failed to get container status \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": rpc error: code = NotFound desc = could not find container \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": container with ID starting with 559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c not found: ID does not exist"
Jul 24 17:42:40.085031 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.084970    2551 scope.go:117] "RemoveContainer" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:40.085239 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.085219    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"} err="failed to get container status \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": rpc error: code = NotFound desc = could not find container \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": container with ID starting with 6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648 not found: ID does not exist"
Jul 24 17:42:40.085288 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.085239    2551 scope.go:117] "RemoveContainer" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.085511 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.085493    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"} err="failed to get container status \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": rpc error: code = NotFound desc = could not find container \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": container with ID starting with 45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65 not found: ID does not exist"
Jul 24 17:42:40.085511 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.085513    2551 scope.go:117] "RemoveContainer" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.085877 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.085846    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"} err="failed to get container status \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": rpc error: code = NotFound desc = could not find container \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": container with ID starting with 53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64 not found: ID does not exist"
Jul 24 17:42:40.085976 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.085877    2551 scope.go:117] "RemoveContainer" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.086600 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.086573    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"} err="failed to get container status \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": rpc error: code = NotFound desc = could not find container \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": container with ID starting with b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe not found: ID does not exist"
Jul 24 17:42:40.086665 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.086602    2551 scope.go:117] "RemoveContainer" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.086879 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.086856    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"} err="failed to get container status \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": rpc error: code = NotFound desc = could not find container \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": container with ID starting with 3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f not found: ID does not exist"
Jul 24 17:42:40.086952 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.086882    2551 scope.go:117] "RemoveContainer" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.087127 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087106    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"} err="failed to get container status \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": rpc error: code = NotFound desc = could not find container \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": container with ID starting with 321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97 not found: ID does not exist"
Jul 24 17:42:40.087184 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087130    2551 scope.go:117] "RemoveContainer" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.087426 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087405    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"} err="failed to get container status \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": rpc error: code = NotFound desc = could not find container \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": container with ID starting with 559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c not found: ID does not exist"
Jul 24 17:42:40.087488 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087427    2551 scope.go:117] "RemoveContainer" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:40.087653 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087631    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"} err="failed to get container status \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": rpc error: code = NotFound desc = could not find container \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": container with ID starting with 6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648 not found: ID does not exist"
Jul 24 17:42:40.087653 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087652    2551 scope.go:117] "RemoveContainer" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.087849 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087833    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"} err="failed to get container status \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": rpc error: code = NotFound desc = could not find container \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": container with ID starting with 45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65 not found: ID does not exist"
Jul 24 17:42:40.087898 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.087850    2551 scope.go:117] "RemoveContainer" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.088054 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088033    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"} err="failed to get container status \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": rpc error: code = NotFound desc = could not find container \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": container with ID starting with 53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64 not found: ID does not exist"
Jul 24 17:42:40.088096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088058    2551 scope.go:117] "RemoveContainer" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.088260 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088240    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"} err="failed to get container status \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": rpc error: code = NotFound desc = could not find container \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": container with ID starting with b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe not found: ID does not exist"
Jul 24 17:42:40.088312 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088261    2551 scope.go:117] "RemoveContainer" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.088502 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088486    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"} err="failed to get container status \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": rpc error: code = NotFound desc = could not find container \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": container with ID starting with 3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f not found: ID does not exist"
Jul 24 17:42:40.088565 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088502    2551 scope.go:117] "RemoveContainer" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.088741 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088723    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"} err="failed to get container status \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": rpc error: code = NotFound desc = could not find container \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": container with ID starting with 321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97 not found: ID does not exist"
Jul 24 17:42:40.088783 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088742    2551 scope.go:117] "RemoveContainer" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.088965 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088944    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"} err="failed to get container status \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": rpc error: code = NotFound desc = could not find container \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": container with ID starting with 559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c not found: ID does not exist"
Jul 24 17:42:40.089008 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.088966    2551 scope.go:117] "RemoveContainer" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:40.089215 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089183    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"} err="failed to get container status \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": rpc error: code = NotFound desc = could not find container \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": container with ID starting with 6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648 not found: ID does not exist"
Jul 24 17:42:40.089215 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089207    2551 scope.go:117] "RemoveContainer" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.089445 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089428    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"} err="failed to get container status \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": rpc error: code = NotFound desc = could not find container \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": container with ID starting with 45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65 not found: ID does not exist"
Jul 24 17:42:40.089492 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089446    2551 scope.go:117] "RemoveContainer" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.089643 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089624    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"} err="failed to get container status \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": rpc error: code = NotFound desc = could not find container \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": container with ID starting with 53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64 not found: ID does not exist"
Jul 24 17:42:40.089706 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089645    2551 scope.go:117] "RemoveContainer" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.089850 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089820    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"} err="failed to get container status \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": rpc error: code = NotFound desc = could not find container \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": container with ID starting with b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe not found: ID does not exist"
Jul 24 17:42:40.089915 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.089852    2551 scope.go:117] "RemoveContainer" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.090077 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090058    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"} err="failed to get container status \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": rpc error: code = NotFound desc = could not find container \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": container with ID starting with 3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f not found: ID does not exist"
Jul 24 17:42:40.090117 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090078    2551 scope.go:117] "RemoveContainer" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.090329 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090300    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"} err="failed to get container status \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": rpc error: code = NotFound desc = could not find container \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": container with ID starting with 321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97 not found: ID does not exist"
Jul 24 17:42:40.090329 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090323    2551 scope.go:117] "RemoveContainer" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.090582 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090563    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"} err="failed to get container status \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": rpc error: code = NotFound desc = could not find container \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": container with ID starting with 559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c not found: ID does not exist"
Jul 24 17:42:40.090641 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090583    2551 scope.go:117] "RemoveContainer" containerID="6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"
Jul 24 17:42:40.090867 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090836    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648"} err="failed to get container status \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": rpc error: code = NotFound desc = could not find container \"6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648\": container with ID starting with 6322c764e070a145ef04c8ee9cc38423e982636ff084a691800bc2a9cce41648 not found: ID does not exist"
Jul 24 17:42:40.090867 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.090863    2551 scope.go:117] "RemoveContainer" containerID="45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"
Jul 24 17:42:40.091091 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.091071    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65"} err="failed to get container status \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": rpc error: code = NotFound desc = could not find container \"45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65\": container with ID starting with 45a0ee7f44ed9f11ac870454572bc34f68ae740006ff54574a69150232d2bf65 not found: ID does not exist"
Jul 24 17:42:40.091161 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.091092    2551 scope.go:117] "RemoveContainer" containerID="53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"
Jul 24 17:42:40.091332 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.091313    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64"} err="failed to get container status \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": rpc error: code = NotFound desc = could not find container \"53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64\": container with ID starting with 53869d439f8f8433e6f5abd2aab3a5c52491220cd04dad835a6c9d568b869e64 not found: ID does not exist"
Jul 24 17:42:40.091412 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.091332    2551 scope.go:117] "RemoveContainer" containerID="b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"
Jul 24 17:42:40.091622 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.091604    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe"} err="failed to get container status \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": rpc error: code = NotFound desc = could not find container \"b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe\": container with ID starting with b6a8489ec9bad0dced35fa43cc0c9aa2a6375d3012bb349868e4e8633e7250fe not found: ID does not exist"
Jul 24 17:42:40.091622 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.091622    2551 scope.go:117] "RemoveContainer" containerID="3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"
Jul 24 17:42:40.093651 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.093620    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f"} err="failed to get container status \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": rpc error: code = NotFound desc = could not find container \"3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f\": container with ID starting with 3d9b15b240ae4817d0d72fc26d59fb88715a0350588357e91143ccf9ed88290f not found: ID does not exist"
Jul 24 17:42:40.093651 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.093640    2551 scope.go:117] "RemoveContainer" containerID="321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"
Jul 24 17:42:40.094240 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.093856    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97"} err="failed to get container status \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": rpc error: code = NotFound desc = could not find container \"321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97\": container with ID starting with 321ffda76fa10b2858f78605f4526d97bd943d57087f2421d3db09400da7fb97 not found: ID does not exist"
Jul 24 17:42:40.094240 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.093884    2551 scope.go:117] "RemoveContainer" containerID="559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"
Jul 24 17:42:40.094240 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.094147    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c"} err="failed to get container status \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": rpc error: code = NotFound desc = could not find container \"559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c\": container with ID starting with 559cb6ff18a008105f70469c3817a2dd4fa344eb1ae602196192f98b980ad39c not found: ID does not exist"
Jul 24 17:42:40.334144 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.334114    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:42:40.342286 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.342260    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:42:40.390565 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390537    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:42:40.390857 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390845    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="config-reloader"
Jul 24 17:42:40.390897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390859    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="config-reloader"
Jul 24 17:42:40.390897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390868    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="init-config-reloader"
Jul 24 17:42:40.390897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390874    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="init-config-reloader"
Jul 24 17:42:40.390897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390881    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="prometheus"
Jul 24 17:42:40.390897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390886    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="prometheus"
Jul 24 17:42:40.390897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390893    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy"
Jul 24 17:42:40.390897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390898    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390905    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-web"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390911    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-web"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390922    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-thanos"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390927    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-thanos"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390932    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="thanos-sidecar"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390937    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="thanos-sidecar"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390981    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-thanos"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390992    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="config-reloader"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.390999    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy-web"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.391005    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="prometheus"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.391010    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="thanos-sidecar"
Jul 24 17:42:40.391096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.391015    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" containerName="kube-rbac-proxy"
Jul 24 17:42:40.396014 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.395990    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.405363 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405320    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 17:42:40.405363 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405341    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 17:42:40.405684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405493    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 17:42:40.405684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405526    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 17:42:40.405684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405535    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 17:42:40.405684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405646    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ms2jm\""
Jul 24 17:42:40.405684 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405675    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 17:42:40.405939 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405700    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 17:42:40.405939 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405784    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 17:42:40.405939 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405832    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 17:42:40.405939 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405871    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 17:42:40.406130 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.405962    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 17:42:40.406247 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.406218    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 17:42:40.406314 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.406284    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ave97vfkeefmo\""
Jul 24 17:42:40.411054 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.411035    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 17:42:40.419424 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.419405    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 17:42:40.435729 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.435693    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:42:40.477026 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.476995    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36fc9504-4e7f-4890-932a-e335c670cfeb-config-out\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477026 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477034    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477226 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477063    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477226 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477098    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477226 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477226 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477209    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-config\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477347 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477233    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477347 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477252    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477347 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477276    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36fc9504-4e7f-4890-932a-e335c670cfeb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477347 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477319    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477347 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477342    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477403    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-web-config\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477429    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5k7\" (UniqueName: \"kubernetes.io/projected/36fc9504-4e7f-4890-932a-e335c670cfeb-kube-api-access-5s5k7\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477461    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477483    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477515    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477536    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477554    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477567    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.477792 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.477598    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578427 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578389    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36fc9504-4e7f-4890-932a-e335c670cfeb-config-out\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578592 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578442    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578592 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578491    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578592 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578519    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578616    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578675    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-config\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578708    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578761 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578770    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36fc9504-4e7f-4890-932a-e335c670cfeb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578809    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578841    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578880    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-web-config\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578904    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5k7\" (UniqueName: \"kubernetes.io/projected/36fc9504-4e7f-4890-932a-e335c670cfeb-kube-api-access-5s5k7\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578924    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.578943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578935    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579263 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.578986    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579263 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.579034    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579263 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.579060    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579263 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.579089    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579263 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.579115    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579263 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.579128    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579263 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.579153    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.579615 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.579594    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.580325 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.580302    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.582281 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.581902    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.582281 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.581942    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.582281 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.581942    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-config\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.582548 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.582527    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.582628 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.582604    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.582676 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.582604    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.583109 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.582963    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36fc9504-4e7f-4890-932a-e335c670cfeb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.583191 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.583132    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.583257 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.583220    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36fc9504-4e7f-4890-932a-e335c670cfeb-config-out\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.583313 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.583267    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/36fc9504-4e7f-4890-932a-e335c670cfeb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.583774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.583751    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.585237 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.585210    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.585237 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.585226    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.585897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.585871    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-web-config\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.585897 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.585889    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36fc9504-4e7f-4890-932a-e335c670cfeb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.593037 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.593018    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5k7\" (UniqueName: \"kubernetes.io/projected/36fc9504-4e7f-4890-932a-e335c670cfeb-kube-api-access-5s5k7\") pod \"prometheus-k8s-0\" (UID: \"36fc9504-4e7f-4890-932a-e335c670cfeb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.705057 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.705007    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:42:40.851416 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:40.851308    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 17:42:40.855232 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:42:40.855207    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fc9504_4e7f_4890_932a_e335c670cfeb.slice/crio-0dd9fabf5183764d2b9255dc813d435abd81540a70f3b9681a4f8c739959d448 WatchSource:0}: Error finding container 0dd9fabf5183764d2b9255dc813d435abd81540a70f3b9681a4f8c739959d448: Status 404 returned error can't find the container with id 0dd9fabf5183764d2b9255dc813d435abd81540a70f3b9681a4f8c739959d448
Jul 24 17:42:41.008945 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:41.008909    2551 generic.go:356] "Generic (PLEG): container finished" podID="36fc9504-4e7f-4890-932a-e335c670cfeb" containerID="b946b138942d8c7834e799c1d09460a2f2229669014dae12fed3ac2794fbc47d" exitCode=0
Jul 24 17:42:41.009370 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:41.009004    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerDied","Data":"b946b138942d8c7834e799c1d09460a2f2229669014dae12fed3ac2794fbc47d"}
Jul 24 17:42:41.009370 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:41.009091    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerStarted","Data":"0dd9fabf5183764d2b9255dc813d435abd81540a70f3b9681a4f8c739959d448"}
Jul 24 17:42:41.301808 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:41.301781    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e292caa-f56d-4eca-9233-88f23ca555d5" path="/var/lib/kubelet/pods/2e292caa-f56d-4eca-9233-88f23ca555d5/volumes"
Jul 24 17:42:42.018111 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:42.018075    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerStarted","Data":"1e26a02cf1d9bd1a11c2e9ce186374dc42cc0f013f5ac9e415b202e7ea3bc770"}
Jul 24 17:42:42.018410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:42.018151    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerStarted","Data":"752f27644ed671d042e01b7139b966116f723e4f22d09b0bb63f7b935280d195"}
Jul 24 17:42:42.018410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:42.018178    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerStarted","Data":"63c956d18138864b5ffa61eab1a49b49b2e50cac4d9e4197063eadffac86d1bc"}
Jul 24 17:42:42.018410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:42.018209    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerStarted","Data":"c85d915ceeb1c08e2d1cb6b69e409b85e1f5a1470738bab682d0ae91cb9f789b"}
Jul 24 17:42:42.018410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:42.018238    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerStarted","Data":"bd5462c6f651b6d6d06382b83db7cc274f753778d4c6bdc4bb94663bd7b5ffea"}
Jul 24 17:42:42.018410 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:42.018265    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"36fc9504-4e7f-4890-932a-e335c670cfeb","Type":"ContainerStarted","Data":"ce3fd202f1c57a03749d3b63ac9b76d61e7477d52b21a8b2956216ee1f2fe768"}
Jul 24 17:42:42.048461 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:42.048414    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.029420286 podStartE2EDuration="2.048396478s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.976192ms" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:42.046156809 +0000 UTC m=+153.163084178" watchObservedRunningTime="2026-07-24 17:42:42.048396478 +0000 UTC m=+153.165323819"
Jul 24 17:42:43.024223 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:43.024183    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l" event={"ID":"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a","Type":"ContainerStarted","Data":"81a99a999c03daf574cba4b110e1c69a728e01aef358870bdae3e005c6d0394f"}
Jul 24 17:42:43.024632 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:43.024255    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l" event={"ID":"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a","Type":"ContainerStarted","Data":"30470d11b5e6b906203032ff4ba4bc4aa53076e62ddc1079864ccf866e3f1ecf"}
Jul 24 17:42:43.024632 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:43.024282    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l" event={"ID":"4403fc70-3600-4f5f-bd5d-3320f9fa9f7a","Type":"ContainerStarted","Data":"70e61a072a18c71c9a5f3c05b922d98deb079ec7521702f50212537bc47cb98e"}
Jul 24 17:42:43.050946 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:43.050889    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dcc569685-tk72l" podStartSLOduration=1.860400641 podStartE2EDuration="4.050872527s" totalImagesPullingTime="2.190471886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:43.049663641 +0000 UTC m=+154.166590981" watchObservedRunningTime="2026-07-24 17:42:43.050872527 +0000 UTC m=+154.167799868"
Jul 24 17:42:45.705723 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:42:45.705684    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:43:40.705627 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:43:40.705592    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:43:40.721934 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:43:40.721904    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:43:41.269005 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:43:41.268976    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 17:45:09.177916 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:09.177884    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:45:09.180777 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:09.180756    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:45:09.184877 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:09.184861    2551 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 17:45:23.374910 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.374877    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jtqkn"]
Jul 24 17:45:23.378153 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.378138    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.380511 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.380487    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 17:45:23.381330 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.381306    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2tlqg\""
Jul 24 17:45:23.381330 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.381322    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 17:45:23.386005 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.385983    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jtqkn"]
Jul 24 17:45:23.422526 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.422497    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bde0cbd9-30fb-41ee-9a22-418b582b082f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jtqkn\" (UID: \"bde0cbd9-30fb-41ee-9a22-418b582b082f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.422646 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.422546    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mbm\" (UniqueName: \"kubernetes.io/projected/bde0cbd9-30fb-41ee-9a22-418b582b082f-kube-api-access-t2mbm\") pod \"cert-manager-webhook-6486ffd997-jtqkn\" (UID: \"bde0cbd9-30fb-41ee-9a22-418b582b082f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.523700 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.523672    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mbm\" (UniqueName: \"kubernetes.io/projected/bde0cbd9-30fb-41ee-9a22-418b582b082f-kube-api-access-t2mbm\") pod \"cert-manager-webhook-6486ffd997-jtqkn\" (UID: \"bde0cbd9-30fb-41ee-9a22-418b582b082f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.523861 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.523769    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bde0cbd9-30fb-41ee-9a22-418b582b082f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jtqkn\" (UID: \"bde0cbd9-30fb-41ee-9a22-418b582b082f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.533335 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.533305    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bde0cbd9-30fb-41ee-9a22-418b582b082f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jtqkn\" (UID: \"bde0cbd9-30fb-41ee-9a22-418b582b082f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.533475 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.533458    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mbm\" (UniqueName: \"kubernetes.io/projected/bde0cbd9-30fb-41ee-9a22-418b582b082f-kube-api-access-t2mbm\") pod \"cert-manager-webhook-6486ffd997-jtqkn\" (UID: \"bde0cbd9-30fb-41ee-9a22-418b582b082f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.706043 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.705952    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:23.837757 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.837722    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jtqkn"]
Jul 24 17:45:23.840887 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:45:23.840857    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbde0cbd9_30fb_41ee_9a22_418b582b082f.slice/crio-7886998d1d0f6c67a3961b1e3c0473aecfc6ec387978821f0320b1b5bcc32df4 WatchSource:0}: Error finding container 7886998d1d0f6c67a3961b1e3c0473aecfc6ec387978821f0320b1b5bcc32df4: Status 404 returned error can't find the container with id 7886998d1d0f6c67a3961b1e3c0473aecfc6ec387978821f0320b1b5bcc32df4
Jul 24 17:45:23.842977 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:23.842958    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:45:24.667140 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.667103    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn" event={"ID":"bde0cbd9-30fb-41ee-9a22-418b582b082f","Type":"ContainerStarted","Data":"7886998d1d0f6c67a3961b1e3c0473aecfc6ec387978821f0320b1b5bcc32df4"}
Jul 24 17:45:24.773630 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.773588    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"]
Jul 24 17:45:24.778196 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.778177    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:24.780480 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.780459    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zklt6\""
Jul 24 17:45:24.786532 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.786505    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"]
Jul 24 17:45:24.836326 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.836286    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-h7jft\" (UID: \"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:24.836513 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.836344    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gjwc\" (UniqueName: \"kubernetes.io/projected/84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e-kube-api-access-6gjwc\") pod \"cert-manager-cainjector-559cdd8c87-h7jft\" (UID: \"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:24.936973 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.936875    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-h7jft\" (UID: \"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:24.936973 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.936966    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gjwc\" (UniqueName: \"kubernetes.io/projected/84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e-kube-api-access-6gjwc\") pod \"cert-manager-cainjector-559cdd8c87-h7jft\" (UID: \"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:24.946573 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.946506    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-h7jft\" (UID: \"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:24.946718 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:24.946621    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gjwc\" (UniqueName: \"kubernetes.io/projected/84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e-kube-api-access-6gjwc\") pod \"cert-manager-cainjector-559cdd8c87-h7jft\" (UID: \"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:25.088285 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:25.088033    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"
Jul 24 17:45:25.257295 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:25.257260    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-h7jft"]
Jul 24 17:45:25.261168 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:45:25.261139    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84ad1ed8_b4fc_47f7_b1e8_c6f0e503953e.slice/crio-fdfa3829a49a1561b0374211be22bc7b500fc81a33ae1f67e741dce34bf24103 WatchSource:0}: Error finding container fdfa3829a49a1561b0374211be22bc7b500fc81a33ae1f67e741dce34bf24103: Status 404 returned error can't find the container with id fdfa3829a49a1561b0374211be22bc7b500fc81a33ae1f67e741dce34bf24103
Jul 24 17:45:25.673576 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:25.673516    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft" event={"ID":"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e","Type":"ContainerStarted","Data":"fdfa3829a49a1561b0374211be22bc7b500fc81a33ae1f67e741dce34bf24103"}
Jul 24 17:45:27.684955 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:27.684921    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn" event={"ID":"bde0cbd9-30fb-41ee-9a22-418b582b082f","Type":"ContainerStarted","Data":"8598626187cff853790c19f5e25fa664adfc4e1a0ae02fb396571266cfc19204"}
Jul 24 17:45:27.685469 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:27.685000    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:27.686676 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:27.686645    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft" event={"ID":"84ad1ed8-b4fc-47f7-b1e8-c6f0e503953e","Type":"ContainerStarted","Data":"1a3970f908a9e0d08f8e95ce8e7226d74929e18d689528e1552e87859b49c873"}
Jul 24 17:45:27.704079 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:27.704019    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn" podStartSLOduration=1.2543257159999999 podStartE2EDuration="4.704001151s" totalImagesPullingTime="3.449675435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:27.702928695 +0000 UTC m=+318.819856035" watchObservedRunningTime="2026-07-24 17:45:27.704001151 +0000 UTC m=+318.820928495"
Jul 24 17:45:27.723916 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:27.723868    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h7jft" podStartSLOduration=1.694408998 podStartE2EDuration="3.723849501s" totalImagesPullingTime="2.029440503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:27.723128128 +0000 UTC m=+318.840055468" watchObservedRunningTime="2026-07-24 17:45:27.723849501 +0000 UTC m=+318.840776844"
Jul 24 17:45:33.693811 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:33.693781    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jtqkn"
Jul 24 17:45:54.879439 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.879402    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"]
Jul 24 17:45:54.887130 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.887113    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:54.893047 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.893025    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 17:45:54.893173 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.893111    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 17:45:54.893173 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.893147    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 17:45:54.895136 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.895115    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ng8ql\""
Jul 24 17:45:54.895248 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.895209    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 17:45:54.914865 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:54.914838    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"]
Jul 24 17:45:55.005132 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.005094    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pj7f\" (UniqueName: \"kubernetes.io/projected/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-kube-api-access-9pj7f\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.005314 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.005141    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.005314 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.005171    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-tmp\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.005314 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.005195    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-manifests\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.005314 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.005210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.106302 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.106268    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pj7f\" (UniqueName: \"kubernetes.io/projected/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-kube-api-access-9pj7f\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.106498 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.106312    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.106498 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.106330    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-tmp\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.106498 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.106461    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-manifests\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.106498 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.106492    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.106791 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.106771    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-manifests\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.108558 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.108531    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-tmp\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.108822 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.108803    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.108884 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.108828    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.124860 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.124833    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pj7f\" (UniqueName: \"kubernetes.io/projected/d0e2302e-afe9-42e8-aa1b-c0b811b514b6-kube-api-access-9pj7f\") pod \"opendatahub-operator-controller-manager-67f579866-wf9ck\" (UID: \"d0e2302e-afe9-42e8-aa1b-c0b811b514b6\") " pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.198986 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.198913    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:45:55.365968 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.365937    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"]
Jul 24 17:45:55.368723 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:45:55.368696    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e2302e_afe9_42e8_aa1b_c0b811b514b6.slice/crio-d2f9aa9b9bfa6296f33bd87a82a2635bd90ac4ce72030807174e2e2778c1930f WatchSource:0}: Error finding container d2f9aa9b9bfa6296f33bd87a82a2635bd90ac4ce72030807174e2e2778c1930f: Status 404 returned error can't find the container with id d2f9aa9b9bfa6296f33bd87a82a2635bd90ac4ce72030807174e2e2778c1930f
Jul 24 17:45:55.798186 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:55.798146    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck" event={"ID":"d0e2302e-afe9-42e8-aa1b-c0b811b514b6","Type":"ContainerStarted","Data":"d2f9aa9b9bfa6296f33bd87a82a2635bd90ac4ce72030807174e2e2778c1930f"}
Jul 24 17:45:58.819582 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:58.819546    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck" event={"ID":"d0e2302e-afe9-42e8-aa1b-c0b811b514b6","Type":"ContainerStarted","Data":"8fb2dfc7810f49d70ad35b14bf18bbfcfdc5803ec707b21e6aeceab3dc9d3040"}
Jul 24 17:45:59.825975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:59.825941    2551 generic.go:356] "Generic (PLEG): container finished" podID="d0e2302e-afe9-42e8-aa1b-c0b811b514b6" containerID="8fb2dfc7810f49d70ad35b14bf18bbfcfdc5803ec707b21e6aeceab3dc9d3040" exitCode=0
Jul 24 17:45:59.826322 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:45:59.826025    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck" event={"ID":"d0e2302e-afe9-42e8-aa1b-c0b811b514b6","Type":"ContainerDied","Data":"8fb2dfc7810f49d70ad35b14bf18bbfcfdc5803ec707b21e6aeceab3dc9d3040"}
Jul 24 17:46:00.832895 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:00.832852    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck" event={"ID":"d0e2302e-afe9-42e8-aa1b-c0b811b514b6","Type":"ContainerStarted","Data":"3fc0237f7a118b50a620702c898c77fedffa433adf719e9c4ae9a2cf1508469c"}
Jul 24 17:46:00.833448 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:00.832948    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:46:00.854500 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:00.854438    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck" podStartSLOduration=2.402118858 podStartE2EDuration="6.854416695s" totalImagesPullingTime="3.379018825s" totalInitContainerRuntime="1.073279012s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:45:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:00.853006246 +0000 UTC m=+351.969933597" watchObservedRunningTime="2026-07-24 17:46:00.854416695 +0000 UTC m=+351.971344037"
Jul 24 17:46:11.840010 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:11.839979    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67f579866-wf9ck"
Jul 24 17:46:13.670406 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.670358    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"]
Jul 24 17:46:13.685632 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.685593    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"]
Jul 24 17:46:13.685865 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.685694    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.688238 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.688217    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 17:46:13.689233 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.689204    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 17:46:13.689436 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.689414    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 17:46:13.689597 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.689451    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hgdt2\""
Jul 24 17:46:13.689597 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.689464    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 17:46:13.689887 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.689872    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 17:46:13.760133 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.760091    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tzj\" (UniqueName: \"kubernetes.io/projected/68acbd59-c8d8-483e-a933-68e1158f73f7-kube-api-access-z2tzj\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.760133 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.760135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68acbd59-c8d8-483e-a933-68e1158f73f7-cert\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.760336 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.760264    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68acbd59-c8d8-483e-a933-68e1158f73f7-metrics-cert\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.760336 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.760302    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68acbd59-c8d8-483e-a933-68e1158f73f7-manager-config\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.861714 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.861674    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68acbd59-c8d8-483e-a933-68e1158f73f7-metrics-cert\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.861884 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.861741    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68acbd59-c8d8-483e-a933-68e1158f73f7-manager-config\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.861884 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.861790    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tzj\" (UniqueName: \"kubernetes.io/projected/68acbd59-c8d8-483e-a933-68e1158f73f7-kube-api-access-z2tzj\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.861884 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.861823    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68acbd59-c8d8-483e-a933-68e1158f73f7-cert\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.862546 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.862516    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/68acbd59-c8d8-483e-a933-68e1158f73f7-manager-config\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.864417 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.864369    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68acbd59-c8d8-483e-a933-68e1158f73f7-cert\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.864576 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.864557    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/68acbd59-c8d8-483e-a933-68e1158f73f7-metrics-cert\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.881480 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.881455    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tzj\" (UniqueName: \"kubernetes.io/projected/68acbd59-c8d8-483e-a933-68e1158f73f7-kube-api-access-z2tzj\") pod \"lws-controller-manager-74c56c9c54-dkzpr\" (UID: \"68acbd59-c8d8-483e-a933-68e1158f73f7\") " pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:13.997067 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:13.996993    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:14.156255 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.156231    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"]
Jul 24 17:46:14.158399 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:46:14.158359    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68acbd59_c8d8_483e_a933_68e1158f73f7.slice/crio-9687f45af3402d6bdea511e94767964e21221cb28f7c9a2760f9b58b706dacf0 WatchSource:0}: Error finding container 9687f45af3402d6bdea511e94767964e21221cb28f7c9a2760f9b58b706dacf0: Status 404 returned error can't find the container with id 9687f45af3402d6bdea511e94767964e21221cb28f7c9a2760f9b58b706dacf0
Jul 24 17:46:14.691002 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.690968    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"]
Jul 24 17:46:14.695624 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.695602    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.697866 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.697842    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 17:46:14.698224 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.698205    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 17:46:14.698467 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.698450    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 17:46:14.698704 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.698684    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 17:46:14.698974 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.698958    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9rqt8\""
Jul 24 17:46:14.705708 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.705681    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"]
Jul 24 17:46:14.770061 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.770022    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj24m\" (UniqueName: \"kubernetes.io/projected/a07eb4bd-43ef-4bcf-b546-5253e607e906-kube-api-access-hj24m\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.770061 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.770066    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07eb4bd-43ef-4bcf-b546-5253e607e906-tmp\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.770249 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.770164    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07eb4bd-43ef-4bcf-b546-5253e607e906-tls-certs\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.871670 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.871361    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07eb4bd-43ef-4bcf-b546-5253e607e906-tls-certs\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.871670 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.871502    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj24m\" (UniqueName: \"kubernetes.io/projected/a07eb4bd-43ef-4bcf-b546-5253e607e906-kube-api-access-hj24m\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.871670 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.871534    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07eb4bd-43ef-4bcf-b546-5253e607e906-tmp\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.874487 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.874451    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07eb4bd-43ef-4bcf-b546-5253e607e906-tmp\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.876669 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.876610    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07eb4bd-43ef-4bcf-b546-5253e607e906-tls-certs\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.882828 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.882775    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj24m\" (UniqueName: \"kubernetes.io/projected/a07eb4bd-43ef-4bcf-b546-5253e607e906-kube-api-access-hj24m\") pod \"kube-auth-proxy-69ddb4868f-vmnsz\" (UID: \"a07eb4bd-43ef-4bcf-b546-5253e607e906\") " pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:14.909108 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:14.909039    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr" event={"ID":"68acbd59-c8d8-483e-a933-68e1158f73f7","Type":"ContainerStarted","Data":"9687f45af3402d6bdea511e94767964e21221cb28f7c9a2760f9b58b706dacf0"}
Jul 24 17:46:15.006757 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:15.006406    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"
Jul 24 17:46:15.204774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:15.204704    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz"]
Jul 24 17:46:15.210517 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:46:15.210478    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07eb4bd_43ef_4bcf_b546_5253e607e906.slice/crio-567d81783b7b5c98fb4840472e9af0d7dd672203de74fb6a43244b0f949e6831 WatchSource:0}: Error finding container 567d81783b7b5c98fb4840472e9af0d7dd672203de74fb6a43244b0f949e6831: Status 404 returned error can't find the container with id 567d81783b7b5c98fb4840472e9af0d7dd672203de74fb6a43244b0f949e6831
Jul 24 17:46:15.919922 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:15.919852    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz" event={"ID":"a07eb4bd-43ef-4bcf-b546-5253e607e906","Type":"ContainerStarted","Data":"567d81783b7b5c98fb4840472e9af0d7dd672203de74fb6a43244b0f949e6831"}
Jul 24 17:46:17.943819 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:17.943788    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr" event={"ID":"68acbd59-c8d8-483e-a933-68e1158f73f7","Type":"ContainerStarted","Data":"3946296ac9f6bbb9461c298be6c23b2c1323fee232d2177ea72dcee38fa6ba34"}
Jul 24 17:46:17.944397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:17.943848    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:46:17.963221 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:17.963148    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr" podStartSLOduration=2.189594584 podStartE2EDuration="4.963128985s" totalImagesPullingTime="2.773534401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:17.960414238 +0000 UTC m=+369.077341580" watchObservedRunningTime="2026-07-24 17:46:17.963128985 +0000 UTC m=+369.080056326"
Jul 24 17:46:18.950862 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:18.950825    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz" event={"ID":"a07eb4bd-43ef-4bcf-b546-5253e607e906","Type":"ContainerStarted","Data":"5d415e9bedb68b41ba3eb7782067f0dcc1235164ff87a8a067ea4bf9a1cb88c8"}
Jul 24 17:46:18.967338 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:18.967273    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69ddb4868f-vmnsz" podStartSLOduration=2.173431774 podStartE2EDuration="4.96725324s" totalImagesPullingTime="2.793821466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:18.966727855 +0000 UTC m=+370.083655192" watchObservedRunningTime="2026-07-24 17:46:18.96725324 +0000 UTC m=+370.084180582"
Jul 24 17:46:28.953393 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:46:28.953336    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74c56c9c54-dkzpr"
Jul 24 17:48:06.097492 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.097459    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"]
Jul 24 17:48:06.100932 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.100916    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.103394 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.103358    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 17:48:06.103691 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.103669    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 17:48:06.103904 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.103891    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g9nm2\""
Jul 24 17:48:06.115796 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.115773    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"]
Jul 24 17:48:06.280572 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.280543    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckr8\" (UniqueName: \"kubernetes.io/projected/7db500af-43e9-417d-be9e-010a31566cf9-kube-api-access-8ckr8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.280731 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.280591    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db500af-43e9-417d-be9e-010a31566cf9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.382048 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.381961    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db500af-43e9-417d-be9e-010a31566cf9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.382222 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.382066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckr8\" (UniqueName: \"kubernetes.io/projected/7db500af-43e9-417d-be9e-010a31566cf9-kube-api-access-8ckr8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.382443 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.382420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db500af-43e9-417d-be9e-010a31566cf9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.391954 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.391933    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckr8\" (UniqueName: \"kubernetes.io/projected/7db500af-43e9-417d-be9e-010a31566cf9-kube-api-access-8ckr8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.410465 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.410443    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:06.549284 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.549253    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"]
Jul 24 17:48:06.552960 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:48:06.552934    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db500af_43e9_417d_be9e_010a31566cf9.slice/crio-c8b45bfc108c974f9da61f5156527f5b27922e42d7220da6d1fe75644f8b7c5e WatchSource:0}: Error finding container c8b45bfc108c974f9da61f5156527f5b27922e42d7220da6d1fe75644f8b7c5e: Status 404 returned error can't find the container with id c8b45bfc108c974f9da61f5156527f5b27922e42d7220da6d1fe75644f8b7c5e
Jul 24 17:48:06.773592 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.773504    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"]
Jul 24 17:48:06.779408 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.777157    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"]
Jul 24 17:48:06.803887 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.803857    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"]
Jul 24 17:48:06.808618 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.808595    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:06.820859 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.820826    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"]
Jul 24 17:48:06.987117 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.987081    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96ggl\" (UniqueName: \"kubernetes.io/projected/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-kube-api-access-96ggl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f47ks\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:06.987291 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:06.987135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f47ks\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:07.088466 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:07.088433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96ggl\" (UniqueName: \"kubernetes.io/projected/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-kube-api-access-96ggl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f47ks\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:07.088640 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:07.088499    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f47ks\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:07.088843 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:07.088823    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f47ks\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:07.098288 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:07.098257    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96ggl\" (UniqueName: \"kubernetes.io/projected/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-kube-api-access-96ggl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-f47ks\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:07.120847 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:07.120821    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:07.318210 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:07.318170    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"]
Jul 24 17:48:07.373992 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:48:07.373916    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a752d5_ecb9_4fe6_8825_0e5b773aa1e7.slice/crio-67524631cc09d6ce9b417233a8244bd4735d6a120de5f52e7f13cd9ac423a9c7 WatchSource:0}: Error finding container 67524631cc09d6ce9b417233a8244bd4735d6a120de5f52e7f13cd9ac423a9c7: Status 404 returned error can't find the container with id 67524631cc09d6ce9b417233a8244bd4735d6a120de5f52e7f13cd9ac423a9c7
Jul 24 17:48:07.443749 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:07.443709    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks" event={"ID":"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7","Type":"ContainerStarted","Data":"67524631cc09d6ce9b417233a8244bd4735d6a120de5f52e7f13cd9ac423a9c7"}
Jul 24 17:48:11.466970 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.466910    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks" event={"ID":"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7","Type":"ContainerStarted","Data":"f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f"}
Jul 24 17:48:11.467417 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.466988    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:11.468495 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.468441    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk" podUID="7db500af-43e9-417d-be9e-010a31566cf9" containerName="manager" containerID="cri-o://3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f" gracePeriod=2
Jul 24 17:48:11.487786 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.487140    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks" podStartSLOduration=2.064155319 podStartE2EDuration="5.4871194s" totalImagesPullingTime="3.422964081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:11.486341242 +0000 UTC m=+482.603268592" watchObservedRunningTime="2026-07-24 17:48:11.4871194 +0000 UTC m=+482.604046740"
Jul 24 17:48:11.488555 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:48:11.488519    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" is forbidden: User \"system:node:ip-10-0-142-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-87.ec2.internal' and this object" podUID="7db500af-43e9-417d-be9e-010a31566cf9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:11.708362 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.708331    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:11.710791 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:48:11.710752    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" is forbidden: User \"system:node:ip-10-0-142-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-87.ec2.internal' and this object" podUID="7db500af-43e9-417d-be9e-010a31566cf9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:11.731148 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.731086    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ckr8\" (UniqueName: \"kubernetes.io/projected/7db500af-43e9-417d-be9e-010a31566cf9-kube-api-access-8ckr8\") pod \"7db500af-43e9-417d-be9e-010a31566cf9\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") "
Jul 24 17:48:11.731148 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.731146    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db500af-43e9-417d-be9e-010a31566cf9-extensions-socket-volume\") pod \"7db500af-43e9-417d-be9e-010a31566cf9\" (UID: \"7db500af-43e9-417d-be9e-010a31566cf9\") "
Jul 24 17:48:11.731438 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.731416    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db500af-43e9-417d-be9e-010a31566cf9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7db500af-43e9-417d-be9e-010a31566cf9" (UID: "7db500af-43e9-417d-be9e-010a31566cf9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:48:11.733333 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.733309    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db500af-43e9-417d-be9e-010a31566cf9-kube-api-access-8ckr8" (OuterVolumeSpecName: "kube-api-access-8ckr8") pod "7db500af-43e9-417d-be9e-010a31566cf9" (UID: "7db500af-43e9-417d-be9e-010a31566cf9"). InnerVolumeSpecName "kube-api-access-8ckr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:48:11.831933 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.831900    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db500af-43e9-417d-be9e-010a31566cf9-extensions-socket-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:48:11.831933 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:11.831932    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ckr8\" (UniqueName: \"kubernetes.io/projected/7db500af-43e9-417d-be9e-010a31566cf9-kube-api-access-8ckr8\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:48:12.475025 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:12.474987    2551 generic.go:356] "Generic (PLEG): container finished" podID="7db500af-43e9-417d-be9e-010a31566cf9" containerID="3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f" exitCode=2
Jul 24 17:48:12.475465 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:12.475044    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:12.475465 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:12.475088    2551 scope.go:117] "RemoveContainer" containerID="3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f"
Jul 24 17:48:12.477330 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:48:12.477308    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" is forbidden: User \"system:node:ip-10-0-142-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-87.ec2.internal' and this object" podUID="7db500af-43e9-417d-be9e-010a31566cf9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:12.484646 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:12.484626    2551 scope.go:117] "RemoveContainer" containerID="3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f"
Jul 24 17:48:12.484934 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:48:12.484916    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f\": container with ID starting with 3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f not found: ID does not exist" containerID="3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f"
Jul 24 17:48:12.484979 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:12.484945    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f"} err="failed to get container status \"3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f\": rpc error: code = NotFound desc = could not find container \"3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f\": container with ID starting with 3c7fe0cd140cd8da77deec5e3c4c756cca943d7f64df26889d05f3fcea9e264f not found: ID does not exist"
Jul 24 17:48:12.485762 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:48:12.485739    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-zprkk\" is forbidden: User \"system:node:ip-10-0-142-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-87.ec2.internal' and this object" podUID="7db500af-43e9-417d-be9e-010a31566cf9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-zprkk"
Jul 24 17:48:13.300857 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:13.300824    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db500af-43e9-417d-be9e-010a31566cf9" path="/var/lib/kubelet/pods/7db500af-43e9-417d-be9e-010a31566cf9/volumes"
Jul 24 17:48:22.477551 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.477521    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:22.775320 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.775242    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"]
Jul 24 17:48:22.775658 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.775633    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db500af-43e9-417d-be9e-010a31566cf9" containerName="manager"
Jul 24 17:48:22.775658 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.775649    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db500af-43e9-417d-be9e-010a31566cf9" containerName="manager"
Jul 24 17:48:22.775810 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.775732    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db500af-43e9-417d-be9e-010a31566cf9" containerName="manager"
Jul 24 17:48:22.783623 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.783603    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:22.791468 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.791448    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"]
Jul 24 17:48:22.841397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.840530    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b315545d-a0ec-4b31-8610-8b0819d1fe00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jqcjx\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:22.841397 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.840589    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrxk\" (UniqueName: \"kubernetes.io/projected/b315545d-a0ec-4b31-8610-8b0819d1fe00-kube-api-access-bwrxk\") pod \"kuadrant-operator-controller-manager-86878c769f-jqcjx\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:22.941421 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.941353    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b315545d-a0ec-4b31-8610-8b0819d1fe00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jqcjx\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:22.941586 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.941448    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxk\" (UniqueName: \"kubernetes.io/projected/b315545d-a0ec-4b31-8610-8b0819d1fe00-kube-api-access-bwrxk\") pod \"kuadrant-operator-controller-manager-86878c769f-jqcjx\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:22.941768 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.941742    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b315545d-a0ec-4b31-8610-8b0819d1fe00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jqcjx\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:22.951757 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:22.951732    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrxk\" (UniqueName: \"kubernetes.io/projected/b315545d-a0ec-4b31-8610-8b0819d1fe00-kube-api-access-bwrxk\") pod \"kuadrant-operator-controller-manager-86878c769f-jqcjx\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:23.094862 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:23.094830    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:23.232107 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:23.232080    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"]
Jul 24 17:48:23.236148 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:48:23.236118    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb315545d_a0ec_4b31_8610_8b0819d1fe00.slice/crio-72fd405c03302271f1fb03aefcf0900e1a070abf950cabbcbd43a87bbb781dfc WatchSource:0}: Error finding container 72fd405c03302271f1fb03aefcf0900e1a070abf950cabbcbd43a87bbb781dfc: Status 404 returned error can't find the container with id 72fd405c03302271f1fb03aefcf0900e1a070abf950cabbcbd43a87bbb781dfc
Jul 24 17:48:23.530949 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:23.530872    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx" event={"ID":"b315545d-a0ec-4b31-8610-8b0819d1fe00","Type":"ContainerStarted","Data":"9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45"}
Jul 24 17:48:23.530949 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:23.530921    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx" event={"ID":"b315545d-a0ec-4b31-8610-8b0819d1fe00","Type":"ContainerStarted","Data":"72fd405c03302271f1fb03aefcf0900e1a070abf950cabbcbd43a87bbb781dfc"}
Jul 24 17:48:23.531322 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:23.530977    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:23.553206 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:23.553129    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx" podStartSLOduration=1.553114726 podStartE2EDuration="1.553114726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:23.551606811 +0000 UTC m=+494.668534154" watchObservedRunningTime="2026-07-24 17:48:23.553114726 +0000 UTC m=+494.670042065"
Jul 24 17:48:34.538570 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.538537    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 17:48:34.588417 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.588368    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"]
Jul 24 17:48:34.588586 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.588567    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks" podUID="51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" containerName="manager" containerID="cri-o://f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f" gracePeriod=10
Jul 24 17:48:34.843150 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.843124    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:34.956991 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.956962    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96ggl\" (UniqueName: \"kubernetes.io/projected/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-kube-api-access-96ggl\") pod \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") "
Jul 24 17:48:34.957155 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.957006    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-extensions-socket-volume\") pod \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\" (UID: \"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7\") "
Jul 24 17:48:34.957571 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.957547    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" (UID: "51a752d5-ecb9-4fe6-8825-0e5b773aa1e7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 17:48:34.959106 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:34.959080    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-kube-api-access-96ggl" (OuterVolumeSpecName: "kube-api-access-96ggl") pod "51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" (UID: "51a752d5-ecb9-4fe6-8825-0e5b773aa1e7"). InnerVolumeSpecName "kube-api-access-96ggl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:48:35.058450 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.058413    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96ggl\" (UniqueName: \"kubernetes.io/projected/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-kube-api-access-96ggl\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:48:35.058450 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.058444    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7-extensions-socket-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:48:35.589440 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.589342    2551 generic.go:356] "Generic (PLEG): container finished" podID="51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" containerID="f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f" exitCode=0
Jul 24 17:48:35.589440 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.589410    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"
Jul 24 17:48:35.589851 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.589405    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks" event={"ID":"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7","Type":"ContainerDied","Data":"f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f"}
Jul 24 17:48:35.589851 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.589521    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks" event={"ID":"51a752d5-ecb9-4fe6-8825-0e5b773aa1e7","Type":"ContainerDied","Data":"67524631cc09d6ce9b417233a8244bd4735d6a120de5f52e7f13cd9ac423a9c7"}
Jul 24 17:48:35.589851 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.589550    2551 scope.go:117] "RemoveContainer" containerID="f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f"
Jul 24 17:48:35.598439 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.598426    2551 scope.go:117] "RemoveContainer" containerID="f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f"
Jul 24 17:48:35.598688 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:48:35.598673    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f\": container with ID starting with f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f not found: ID does not exist" containerID="f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f"
Jul 24 17:48:35.598736 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.598698    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f"} err="failed to get container status \"f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f\": rpc error: code = NotFound desc = could not find container \"f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f\": container with ID starting with f69ba1b71cfb7c29c1501051e759720cc739e573fa106146d678c3375081322f not found: ID does not exist"
Jul 24 17:48:35.630145 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.630123    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"]
Jul 24 17:48:35.631627 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:35.631604    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-f47ks"]
Jul 24 17:48:37.300675 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:37.300636    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" path="/var/lib/kubelet/pods/51a752d5-ecb9-4fe6-8825-0e5b773aa1e7/volumes"
Jul 24 17:48:54.918366 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.918334    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:48:54.918921 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.918770    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" containerName="manager"
Jul 24 17:48:54.918921 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.918788    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" containerName="manager"
Jul 24 17:48:54.918921 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.918879    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a752d5-ecb9-4fe6-8825-0e5b773aa1e7" containerName="manager"
Jul 24 17:48:54.922822 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.922804    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:54.924999 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.924977    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 17:48:54.925137 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.925119    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gd6n5\""
Jul 24 17:48:54.928316 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.928295    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4feebff-b7fe-4882-9e10-a5d80d993db4-config-file\") pod \"limitador-limitador-54d6d988d5-9gj8l\" (UID: \"d4feebff-b7fe-4882-9e10-a5d80d993db4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:54.928513 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.928491    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxbp\" (UniqueName: \"kubernetes.io/projected/d4feebff-b7fe-4882-9e10-a5d80d993db4-kube-api-access-7mxbp\") pod \"limitador-limitador-54d6d988d5-9gj8l\" (UID: \"d4feebff-b7fe-4882-9e10-a5d80d993db4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:54.931259 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.931241    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:48:54.964278 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:54.964243    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:48:55.029928 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:55.029902    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4feebff-b7fe-4882-9e10-a5d80d993db4-config-file\") pod \"limitador-limitador-54d6d988d5-9gj8l\" (UID: \"d4feebff-b7fe-4882-9e10-a5d80d993db4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:55.030081 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:55.029939    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxbp\" (UniqueName: \"kubernetes.io/projected/d4feebff-b7fe-4882-9e10-a5d80d993db4-kube-api-access-7mxbp\") pod \"limitador-limitador-54d6d988d5-9gj8l\" (UID: \"d4feebff-b7fe-4882-9e10-a5d80d993db4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:55.030559 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:55.030541    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4feebff-b7fe-4882-9e10-a5d80d993db4-config-file\") pod \"limitador-limitador-54d6d988d5-9gj8l\" (UID: \"d4feebff-b7fe-4882-9e10-a5d80d993db4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:55.039305 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:55.039272    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxbp\" (UniqueName: \"kubernetes.io/projected/d4feebff-b7fe-4882-9e10-a5d80d993db4-kube-api-access-7mxbp\") pod \"limitador-limitador-54d6d988d5-9gj8l\" (UID: \"d4feebff-b7fe-4882-9e10-a5d80d993db4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:55.234167 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:55.234073    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:55.371218 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:55.371184    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:48:55.374487 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:48:55.374460    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4feebff_b7fe_4882_9e10_a5d80d993db4.slice/crio-1d2d6dcea2d35095fbc00776a75a8b3398cdcaa844946a1f3d134c582b3243a0 WatchSource:0}: Error finding container 1d2d6dcea2d35095fbc00776a75a8b3398cdcaa844946a1f3d134c582b3243a0: Status 404 returned error can't find the container with id 1d2d6dcea2d35095fbc00776a75a8b3398cdcaa844946a1f3d134c582b3243a0
Jul 24 17:48:55.684523 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:55.684483    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l" event={"ID":"d4feebff-b7fe-4882-9e10-a5d80d993db4","Type":"ContainerStarted","Data":"1d2d6dcea2d35095fbc00776a75a8b3398cdcaa844946a1f3d134c582b3243a0"}
Jul 24 17:48:58.702665 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:58.702626    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l" event={"ID":"d4feebff-b7fe-4882-9e10-a5d80d993db4","Type":"ContainerStarted","Data":"4ef8d565ea7fb85183089ca99c3b4df55b0ac4eb4ab56507a099778d6c4feda6"}
Jul 24 17:48:58.703096 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:58.702718    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:48:58.719396 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:48:58.719302    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l" podStartSLOduration=2.196195544 podStartE2EDuration="4.719287202s" totalImagesPullingTime="2.523091658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:58.717251695 +0000 UTC m=+529.834179035" watchObservedRunningTime="2026-07-24 17:48:58.719287202 +0000 UTC m=+529.836214542"
Jul 24 17:49:09.708577 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:09.708542    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"
Jul 24 17:49:24.468866 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.468828    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bc66bb777-ngs7s"]
Jul 24 17:49:24.473919 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.473900    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.476890 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.476868    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cnq2w\""
Jul 24 17:49:24.477017 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.476920    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 17:49:24.480167 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.480144    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bc66bb777-ngs7s"]
Jul 24 17:49:24.592598 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.592561    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd59z\" (UniqueName: \"kubernetes.io/projected/3ea9e844-7ce6-4c7d-855b-f68922877b02-kube-api-access-rd59z\") pod \"authorino-bc66bb777-ngs7s\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") " pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.592769 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.592615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ea9e844-7ce6-4c7d-855b-f68922877b02-tls-cert\") pod \"authorino-bc66bb777-ngs7s\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") " pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.693786 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.693751    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59z\" (UniqueName: \"kubernetes.io/projected/3ea9e844-7ce6-4c7d-855b-f68922877b02-kube-api-access-rd59z\") pod \"authorino-bc66bb777-ngs7s\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") " pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.693979 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.693817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ea9e844-7ce6-4c7d-855b-f68922877b02-tls-cert\") pod \"authorino-bc66bb777-ngs7s\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") " pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.696337 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.696310    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ea9e844-7ce6-4c7d-855b-f68922877b02-tls-cert\") pod \"authorino-bc66bb777-ngs7s\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") " pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.702186 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.702166    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd59z\" (UniqueName: \"kubernetes.io/projected/3ea9e844-7ce6-4c7d-855b-f68922877b02-kube-api-access-rd59z\") pod \"authorino-bc66bb777-ngs7s\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") " pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.784296 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.784209    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:49:24.916264 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:24.916237    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bc66bb777-ngs7s"]
Jul 24 17:49:24.918939 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:49:24.918912    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ea9e844_7ce6_4c7d_855b_f68922877b02.slice/crio-2ae641d1634041cbfdc55b1fc925006916a2c345c84ecfcb03508f5f9378daae WatchSource:0}: Error finding container 2ae641d1634041cbfdc55b1fc925006916a2c345c84ecfcb03508f5f9378daae: Status 404 returned error can't find the container with id 2ae641d1634041cbfdc55b1fc925006916a2c345c84ecfcb03508f5f9378daae
Jul 24 17:49:25.851184 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:25.851144    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc66bb777-ngs7s" event={"ID":"3ea9e844-7ce6-4c7d-855b-f68922877b02","Type":"ContainerStarted","Data":"2ae641d1634041cbfdc55b1fc925006916a2c345c84ecfcb03508f5f9378daae"}
Jul 24 17:49:27.864160 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:27.864115    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc66bb777-ngs7s" event={"ID":"3ea9e844-7ce6-4c7d-855b-f68922877b02","Type":"ContainerStarted","Data":"c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b"}
Jul 24 17:49:27.888165 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:49:27.888108    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bc66bb777-ngs7s" podStartSLOduration=1.044896695 podStartE2EDuration="3.888088414s" totalImagesPullingTime="2.843191719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:49:27.885655044 +0000 UTC m=+559.002582385" watchObservedRunningTime="2026-07-24 17:49:27.888088414 +0000 UTC m=+559.005015754"
Jul 24 17:50:08.579998 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:50:08.579959    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:50:09.212529 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:50:09.212505    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:50:09.212700 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:50:09.212565    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:50:24.563269 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:50:24.563232    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:50:36.159045 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:50:36.158964    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:50:47.608323 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:50:47.608281    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:50:59.860341 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:50:59.860299    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:51:23.574618 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:23.574578    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:51:46.390582 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.390531    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"]
Jul 24 17:51:46.392787 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.392766    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.395635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.395614    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 17:51:46.395770 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.395631    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k52rs\""
Jul 24 17:51:46.396420 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.396399    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 17:51:46.396510 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.396466    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 24 17:51:46.405874 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.405851    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"]
Jul 24 17:51:46.483987 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.483952    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66l5b\" (UniqueName: \"kubernetes.io/projected/b412203b-3299-4a66-b0da-fe81a7edbbdd-kube-api-access-66l5b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.484152 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.484000    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.484152 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.484019    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.484152 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.484106    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.484262 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.484151    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.484262 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.484213    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b412203b-3299-4a66-b0da-fe81a7edbbdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.584943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.584909    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.584943 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.584945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.585181 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.585001    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.585181 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.585042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.585181 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.585088    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b412203b-3299-4a66-b0da-fe81a7edbbdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.585181 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.585158    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66l5b\" (UniqueName: \"kubernetes.io/projected/b412203b-3299-4a66-b0da-fe81a7edbbdd-kube-api-access-66l5b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.585429 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.585364    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.585583 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.585520    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.585583 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.585543    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.587657 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.587633    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b412203b-3299-4a66-b0da-fe81a7edbbdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.587805 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.587783    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b412203b-3299-4a66-b0da-fe81a7edbbdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.593940 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.593910    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66l5b\" (UniqueName: \"kubernetes.io/projected/b412203b-3299-4a66-b0da-fe81a7edbbdd-kube-api-access-66l5b\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l\" (UID: \"b412203b-3299-4a66-b0da-fe81a7edbbdd\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.702955 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.702871    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:46.841127 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.841087    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"]
Jul 24 17:51:46.843399 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:51:46.843348    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb412203b_3299_4a66_b0da_fe81a7edbbdd.slice/crio-7fda69f2d7b2dfc7c508edb27dc154bed7d96eae24b0225e5075df78d6dc5289 WatchSource:0}: Error finding container 7fda69f2d7b2dfc7c508edb27dc154bed7d96eae24b0225e5075df78d6dc5289: Status 404 returned error can't find the container with id 7fda69f2d7b2dfc7c508edb27dc154bed7d96eae24b0225e5075df78d6dc5289
Jul 24 17:51:46.844647 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:46.844629    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 17:51:47.513470 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:47.513430    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l" event={"ID":"b412203b-3299-4a66-b0da-fe81a7edbbdd","Type":"ContainerStarted","Data":"7fda69f2d7b2dfc7c508edb27dc154bed7d96eae24b0225e5075df78d6dc5289"}
Jul 24 17:51:48.525135 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:48.525097    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l" event={"ID":"b412203b-3299-4a66-b0da-fe81a7edbbdd","Type":"ContainerStarted","Data":"bf22eac58348876f91be1071b2dd17ecd9f1fb7ec223147974c100ccb438c1d0"}
Jul 24 17:51:48.547294 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:48.547244    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l" podStartSLOduration=1.49529645 podStartE2EDuration="2.547229978s" totalImagesPullingTime="1.051933528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:51:48.546029394 +0000 UTC m=+699.662956735" watchObservedRunningTime="2026-07-24 17:51:48.547229978 +0000 UTC m=+699.664157317"
Jul 24 17:51:49.164299 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:49.164264    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:51:56.703774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:56.703737    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:56.704197 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:56.703838    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:56.716825 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:56.716791    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:51:57.583413 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:51:57.583363    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l"
Jul 24 17:53:41.146774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.146678    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bc66bb777-ngs7s"]
Jul 24 17:53:41.147365 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.146866    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bc66bb777-ngs7s" podUID="3ea9e844-7ce6-4c7d-855b-f68922877b02" containerName="authorino" containerID="cri-o://c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b" gracePeriod=30
Jul 24 17:53:41.406699 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.406627    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:53:41.536745 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.536706    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ea9e844-7ce6-4c7d-855b-f68922877b02-tls-cert\") pod \"3ea9e844-7ce6-4c7d-855b-f68922877b02\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") "
Jul 24 17:53:41.536745 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.536750    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd59z\" (UniqueName: \"kubernetes.io/projected/3ea9e844-7ce6-4c7d-855b-f68922877b02-kube-api-access-rd59z\") pod \"3ea9e844-7ce6-4c7d-855b-f68922877b02\" (UID: \"3ea9e844-7ce6-4c7d-855b-f68922877b02\") "
Jul 24 17:53:41.538942 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.538908    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea9e844-7ce6-4c7d-855b-f68922877b02-kube-api-access-rd59z" (OuterVolumeSpecName: "kube-api-access-rd59z") pod "3ea9e844-7ce6-4c7d-855b-f68922877b02" (UID: "3ea9e844-7ce6-4c7d-855b-f68922877b02"). InnerVolumeSpecName "kube-api-access-rd59z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:53:41.548635 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.548607    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea9e844-7ce6-4c7d-855b-f68922877b02-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3ea9e844-7ce6-4c7d-855b-f68922877b02" (UID: "3ea9e844-7ce6-4c7d-855b-f68922877b02"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:53:41.638439 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.638403    2551 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ea9e844-7ce6-4c7d-855b-f68922877b02-tls-cert\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:53:41.638439 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:41.638434    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd59z\" (UniqueName: \"kubernetes.io/projected/3ea9e844-7ce6-4c7d-855b-f68922877b02-kube-api-access-rd59z\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:53:42.062806 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.062769    2551 generic.go:356] "Generic (PLEG): container finished" podID="3ea9e844-7ce6-4c7d-855b-f68922877b02" containerID="c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b" exitCode=0
Jul 24 17:53:42.062806 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.062804    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc66bb777-ngs7s" event={"ID":"3ea9e844-7ce6-4c7d-855b-f68922877b02","Type":"ContainerDied","Data":"c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b"}
Jul 24 17:53:42.063055 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.062833    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc66bb777-ngs7s" event={"ID":"3ea9e844-7ce6-4c7d-855b-f68922877b02","Type":"ContainerDied","Data":"2ae641d1634041cbfdc55b1fc925006916a2c345c84ecfcb03508f5f9378daae"}
Jul 24 17:53:42.063055 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.062849    2551 scope.go:117] "RemoveContainer" containerID="c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b"
Jul 24 17:53:42.063055 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.062874    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc66bb777-ngs7s"
Jul 24 17:53:42.071835 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.071816    2551 scope.go:117] "RemoveContainer" containerID="c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b"
Jul 24 17:53:42.072098 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:53:42.072077    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b\": container with ID starting with c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b not found: ID does not exist" containerID="c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b"
Jul 24 17:53:42.072160 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.072108    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b"} err="failed to get container status \"c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b\": rpc error: code = NotFound desc = could not find container \"c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b\": container with ID starting with c6d1732fba1463051fa9dc49e4c0353ad2337245e210807f061bf59b32362e3b not found: ID does not exist"
Jul 24 17:53:42.084168 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.084145    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bc66bb777-ngs7s"]
Jul 24 17:53:42.085790 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:42.085770    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bc66bb777-ngs7s"]
Jul 24 17:53:43.300532 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:53:43.300498    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea9e844-7ce6-4c7d-855b-f68922877b02" path="/var/lib/kubelet/pods/3ea9e844-7ce6-4c7d-855b-f68922877b02/volumes"
Jul 24 17:54:12.045305 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:54:12.045256    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:54:22.948925 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:54:22.948885    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:54:33.449258 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:54:33.449224    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:55:09.242722 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:09.242649    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:55:09.245186 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:09.245162    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 17:55:32.300185 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.300142    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d669b67d-2llf2"]
Jul 24 17:55:32.302725 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.300540    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea9e844-7ce6-4c7d-855b-f68922877b02" containerName="authorino"
Jul 24 17:55:32.302725 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.300553    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea9e844-7ce6-4c7d-855b-f68922877b02" containerName="authorino"
Jul 24 17:55:32.302725 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.300628    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea9e844-7ce6-4c7d-855b-f68922877b02" containerName="authorino"
Jul 24 17:55:32.303728 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.303712    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.305966 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.305947    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:55:32.306821 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.306801    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:55:32.306918 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.306856    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z2qkg\""
Jul 24 17:55:32.312004 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.311974    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d669b67d-2llf2"]
Jul 24 17:55:32.329651 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.329618    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-webhook-cert\") pod \"maas-controller-84d669b67d-2llf2\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") " pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.329801 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.329728    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78p2v\" (UniqueName: \"kubernetes.io/projected/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-kube-api-access-78p2v\") pod \"maas-controller-84d669b67d-2llf2\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") " pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.430213 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.430177    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-webhook-cert\") pod \"maas-controller-84d669b67d-2llf2\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") " pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.430421 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.430300    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78p2v\" (UniqueName: \"kubernetes.io/projected/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-kube-api-access-78p2v\") pod \"maas-controller-84d669b67d-2llf2\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") " pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.432820 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.432802    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-webhook-cert\") pod \"maas-controller-84d669b67d-2llf2\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") " pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.438334 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.438310    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78p2v\" (UniqueName: \"kubernetes.io/projected/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-kube-api-access-78p2v\") pod \"maas-controller-84d669b67d-2llf2\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") " pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.615553 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.615511    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:32.751774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:32.751750    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d669b67d-2llf2"]
Jul 24 17:55:32.754179 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:55:32.754147    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14770ea8_1acc_43b1_8d3c_d5f99d4ddc62.slice/crio-d23aeba585c309f198edb1f06c7fe33c17c3cdd97fb52df9ce28e9c8a5898002 WatchSource:0}: Error finding container d23aeba585c309f198edb1f06c7fe33c17c3cdd97fb52df9ce28e9c8a5898002: Status 404 returned error can't find the container with id d23aeba585c309f198edb1f06c7fe33c17c3cdd97fb52df9ce28e9c8a5898002
Jul 24 17:55:33.586450 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:33.586411    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d669b67d-2llf2" event={"ID":"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62","Type":"ContainerStarted","Data":"d23aeba585c309f198edb1f06c7fe33c17c3cdd97fb52df9ce28e9c8a5898002"}
Jul 24 17:55:35.598211 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:35.598111    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d669b67d-2llf2" event={"ID":"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62","Type":"ContainerStarted","Data":"25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1"}
Jul 24 17:55:35.598789 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:35.598296    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:35.614975 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:35.614919    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d669b67d-2llf2" podStartSLOduration=1.065027489 podStartE2EDuration="3.61490351s" totalImagesPullingTime="2.549876021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:35.614209537 +0000 UTC m=+926.731136880" watchObservedRunningTime="2026-07-24 17:55:35.61490351 +0000 UTC m=+926.731830887"
Jul 24 17:55:46.604936 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:46.604902    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:51.382465 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.382420    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d669b67d-2llf2"]
Jul 24 17:55:51.382969 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.382689    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84d669b67d-2llf2" podUID="14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" containerName="manager" containerID="cri-o://25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1" gracePeriod=30
Jul 24 17:55:51.623076 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.623051    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:51.673457 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.673361    2551 generic.go:356] "Generic (PLEG): container finished" podID="14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" containerID="25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1" exitCode=0
Jul 24 17:55:51.673457 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.673448    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d669b67d-2llf2"
Jul 24 17:55:51.673630 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.673462    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d669b67d-2llf2" event={"ID":"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62","Type":"ContainerDied","Data":"25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1"}
Jul 24 17:55:51.673630 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.673515    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d669b67d-2llf2" event={"ID":"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62","Type":"ContainerDied","Data":"d23aeba585c309f198edb1f06c7fe33c17c3cdd97fb52df9ce28e9c8a5898002"}
Jul 24 17:55:51.673630 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.673538    2551 scope.go:117] "RemoveContainer" containerID="25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1"
Jul 24 17:55:51.682166 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.682150    2551 scope.go:117] "RemoveContainer" containerID="25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1"
Jul 24 17:55:51.682428 ip-10-0-142-87 kubenswrapper[2551]: E0724 17:55:51.682409    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1\": container with ID starting with 25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1 not found: ID does not exist" containerID="25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1"
Jul 24 17:55:51.682487 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.682440    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1"} err="failed to get container status \"25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1\": rpc error: code = NotFound desc = could not find container \"25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1\": container with ID starting with 25e774561b8965990b7a67a9740256b2ceadfa6c5c5f1ebc06c2bcad7aa4f0d1 not found: ID does not exist"
Jul 24 17:55:51.711247 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.711227    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-webhook-cert\") pod \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") "
Jul 24 17:55:51.711357 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.711293    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78p2v\" (UniqueName: \"kubernetes.io/projected/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-kube-api-access-78p2v\") pod \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\" (UID: \"14770ea8-1acc-43b1-8d3c-d5f99d4ddc62\") "
Jul 24 17:55:51.713517 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.713489    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-kube-api-access-78p2v" (OuterVolumeSpecName: "kube-api-access-78p2v") pod "14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" (UID: "14770ea8-1acc-43b1-8d3c-d5f99d4ddc62"). InnerVolumeSpecName "kube-api-access-78p2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 17:55:51.713606 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.713513    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" (UID: "14770ea8-1acc-43b1-8d3c-d5f99d4ddc62"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 17:55:51.812205 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.812172    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78p2v\" (UniqueName: \"kubernetes.io/projected/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-kube-api-access-78p2v\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:55:51.812205 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.812199    2551 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62-webhook-cert\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 17:55:51.994192 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.994164    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d669b67d-2llf2"]
Jul 24 17:55:51.998055 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:51.998031    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84d669b67d-2llf2"]
Jul 24 17:55:52.916357 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.916322    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d669b67d-2ckbr"]
Jul 24 17:55:52.916774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.916741    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" containerName="manager"
Jul 24 17:55:52.916774 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.916752    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" containerName="manager"
Jul 24 17:55:52.916841 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.916828    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" containerName="manager"
Jul 24 17:55:52.920285 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.920269    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:52.925231 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.925201    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z2qkg\""
Jul 24 17:55:52.925846 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.925822    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 17:55:52.925981 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.925891    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 17:55:52.952725 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:52.952694    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d669b67d-2ckbr"]
Jul 24 17:55:53.021887 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.021853    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89df1f4e-da55-454d-9573-245ef2699eec-webhook-cert\") pod \"maas-controller-84d669b67d-2ckbr\" (UID: \"89df1f4e-da55-454d-9573-245ef2699eec\") " pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:53.022032 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.021919    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g9j\" (UniqueName: \"kubernetes.io/projected/89df1f4e-da55-454d-9573-245ef2699eec-kube-api-access-b7g9j\") pod \"maas-controller-84d669b67d-2ckbr\" (UID: \"89df1f4e-da55-454d-9573-245ef2699eec\") " pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:53.123292 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.123260    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89df1f4e-da55-454d-9573-245ef2699eec-webhook-cert\") pod \"maas-controller-84d669b67d-2ckbr\" (UID: \"89df1f4e-da55-454d-9573-245ef2699eec\") " pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:53.123509 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.123330    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g9j\" (UniqueName: \"kubernetes.io/projected/89df1f4e-da55-454d-9573-245ef2699eec-kube-api-access-b7g9j\") pod \"maas-controller-84d669b67d-2ckbr\" (UID: \"89df1f4e-da55-454d-9573-245ef2699eec\") " pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:53.125830 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.125803    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89df1f4e-da55-454d-9573-245ef2699eec-webhook-cert\") pod \"maas-controller-84d669b67d-2ckbr\" (UID: \"89df1f4e-da55-454d-9573-245ef2699eec\") " pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:53.131244 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.131218    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g9j\" (UniqueName: \"kubernetes.io/projected/89df1f4e-da55-454d-9573-245ef2699eec-kube-api-access-b7g9j\") pod \"maas-controller-84d669b67d-2ckbr\" (UID: \"89df1f4e-da55-454d-9573-245ef2699eec\") " pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:53.231573 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.231494    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:53.301891 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.301863    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14770ea8-1acc-43b1-8d3c-d5f99d4ddc62" path="/var/lib/kubelet/pods/14770ea8-1acc-43b1-8d3c-d5f99d4ddc62/volumes"
Jul 24 17:55:53.369855 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.369828    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d669b67d-2ckbr"]
Jul 24 17:55:53.372215 ip-10-0-142-87 kubenswrapper[2551]: W0724 17:55:53.372186    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89df1f4e_da55_454d_9573_245ef2699eec.slice/crio-9fbcda4aa9253a3072f39d853c20ec1637dd5ce700d6c4ebcf490d76778611e8 WatchSource:0}: Error finding container 9fbcda4aa9253a3072f39d853c20ec1637dd5ce700d6c4ebcf490d76778611e8: Status 404 returned error can't find the container with id 9fbcda4aa9253a3072f39d853c20ec1637dd5ce700d6c4ebcf490d76778611e8
Jul 24 17:55:53.684909 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:53.684872    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d669b67d-2ckbr" event={"ID":"89df1f4e-da55-454d-9573-245ef2699eec","Type":"ContainerStarted","Data":"9fbcda4aa9253a3072f39d853c20ec1637dd5ce700d6c4ebcf490d76778611e8"}
Jul 24 17:55:54.691666 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:54.691631    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d669b67d-2ckbr" event={"ID":"89df1f4e-da55-454d-9573-245ef2699eec","Type":"ContainerStarted","Data":"1a1a81d4a7408d395e04c89666080978a99ef2c495d05374e620f88c3f77513d"}
Jul 24 17:55:54.692046 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:54.691723    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:55:54.709042 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:55:54.708998    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d669b67d-2ckbr" podStartSLOduration=2.363012813 podStartE2EDuration="2.708981957s" totalImagesPullingTime="345.969144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:55:54.707289782 +0000 UTC m=+945.824217122" watchObservedRunningTime="2026-07-24 17:55:54.708981957 +0000 UTC m=+945.825909287"
Jul 24 17:56:05.699024 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:56:05.698991    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d669b67d-2ckbr"
Jul 24 17:56:16.440223 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:56:16.440187    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:56:25.148893 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:56:25.148857    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:56:57.552157 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:56:57.552122    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:57:06.549060 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:57:06.549020    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:57:14.545544 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:57:14.545506    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:57:25.550368 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:57:25.550330    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:57:33.954092 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:57:33.954055    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:57:44.445575 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:57:44.445539    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:57:57.758289 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:57:57.758253    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:58:05.347759 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:58:05.347678    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:58:28.749786 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:58:28.749738    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:58:38.947551 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:58:38.947512    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:59:04.247492 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:59:04.247459    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:59:16.243891 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:59:16.243853    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:59:32.552640 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:59:32.552555    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:59:41.766745 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:59:41.766704    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 17:59:51.157038 ip-10-0-142-87 kubenswrapper[2551]: I0724 17:59:51.156998    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:00:00.946898 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:00.946854    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:00:08.752717 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:08.752680    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:00:09.282473 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:09.282446    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:00:09.284278 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:09.284257    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:00:36.853431 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:36.853396    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"]
Jul 24 18:00:36.855909 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:36.853600    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx" podUID="b315545d-a0ec-4b31-8610-8b0819d1fe00" containerName="manager" containerID="cri-o://9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45" gracePeriod=10
Jul 24 18:00:39.801516 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:39.801490    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 18:00:39.832650 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:39.832623    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b315545d-a0ec-4b31-8610-8b0819d1fe00-extensions-socket-volume\") pod \"b315545d-a0ec-4b31-8610-8b0819d1fe00\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") "
Jul 24 18:00:39.832827 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:39.832676    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwrxk\" (UniqueName: \"kubernetes.io/projected/b315545d-a0ec-4b31-8610-8b0819d1fe00-kube-api-access-bwrxk\") pod \"b315545d-a0ec-4b31-8610-8b0819d1fe00\" (UID: \"b315545d-a0ec-4b31-8610-8b0819d1fe00\") "
Jul 24 18:00:39.833075 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:39.833046    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b315545d-a0ec-4b31-8610-8b0819d1fe00-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b315545d-a0ec-4b31-8610-8b0819d1fe00" (UID: "b315545d-a0ec-4b31-8610-8b0819d1fe00"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:00:39.835116 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:39.835087    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b315545d-a0ec-4b31-8610-8b0819d1fe00-kube-api-access-bwrxk" (OuterVolumeSpecName: "kube-api-access-bwrxk") pod "b315545d-a0ec-4b31-8610-8b0819d1fe00" (UID: "b315545d-a0ec-4b31-8610-8b0819d1fe00"). InnerVolumeSpecName "kube-api-access-bwrxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:00:39.934029 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:39.933938    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b315545d-a0ec-4b31-8610-8b0819d1fe00-extensions-socket-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:00:39.934029 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:39.933986    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrxk\" (UniqueName: \"kubernetes.io/projected/b315545d-a0ec-4b31-8610-8b0819d1fe00-kube-api-access-bwrxk\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:00:40.048205 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.048173    2551 generic.go:356] "Generic (PLEG): container finished" podID="b315545d-a0ec-4b31-8610-8b0819d1fe00" containerID="9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45" exitCode=0
Jul 24 18:00:40.048362 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.048216    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"
Jul 24 18:00:40.048362 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.048217    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx" event={"ID":"b315545d-a0ec-4b31-8610-8b0819d1fe00","Type":"ContainerDied","Data":"9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45"}
Jul 24 18:00:40.048362 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.048274    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx" event={"ID":"b315545d-a0ec-4b31-8610-8b0819d1fe00","Type":"ContainerDied","Data":"72fd405c03302271f1fb03aefcf0900e1a070abf950cabbcbd43a87bbb781dfc"}
Jul 24 18:00:40.048362 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.048295    2551 scope.go:117] "RemoveContainer" containerID="9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45"
Jul 24 18:00:40.058306 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.058291    2551 scope.go:117] "RemoveContainer" containerID="9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45"
Jul 24 18:00:40.058580 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:00:40.058559    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45\": container with ID starting with 9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45 not found: ID does not exist" containerID="9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45"
Jul 24 18:00:40.058652 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.058595    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45"} err="failed to get container status \"9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45\": rpc error: code = NotFound desc = could not find container \"9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45\": container with ID starting with 9e8b029f486b29b155fd31e88142fb111fa167dd994224944008de299f45ea45 not found: ID does not exist"
Jul 24 18:00:40.070084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.070062    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"]
Jul 24 18:00:40.073585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:40.073565    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jqcjx"]
Jul 24 18:00:41.299922 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:00:41.299887    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b315545d-a0ec-4b31-8610-8b0819d1fe00" path="/var/lib/kubelet/pods/b315545d-a0ec-4b31-8610-8b0819d1fe00/volumes"
Jul 24 18:01:38.137622 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.137586    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"]
Jul 24 18:01:38.138021 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.137975    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b315545d-a0ec-4b31-8610-8b0819d1fe00" containerName="manager"
Jul 24 18:01:38.138021 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.137986    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b315545d-a0ec-4b31-8610-8b0819d1fe00" containerName="manager"
Jul 24 18:01:38.138094 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.138054    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b315545d-a0ec-4b31-8610-8b0819d1fe00" containerName="manager"
Jul 24 18:01:38.141176 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.141159    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.145096 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.145076    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g9nm2\""
Jul 24 18:01:38.152928 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.152905    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"]
Jul 24 18:01:38.255332 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.255289    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vdmx5\" (UID: \"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.255545 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.255350    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7wd\" (UniqueName: \"kubernetes.io/projected/2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9-kube-api-access-kr7wd\") pod \"kuadrant-operator-controller-manager-86878c769f-vdmx5\" (UID: \"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.356530 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.356487    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vdmx5\" (UID: \"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.356730 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.356578    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7wd\" (UniqueName: \"kubernetes.io/projected/2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9-kube-api-access-kr7wd\") pod \"kuadrant-operator-controller-manager-86878c769f-vdmx5\" (UID: \"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.356900 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.356878    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vdmx5\" (UID: \"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.383524 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.383500    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7wd\" (UniqueName: \"kubernetes.io/projected/2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9-kube-api-access-kr7wd\") pod \"kuadrant-operator-controller-manager-86878c769f-vdmx5\" (UID: \"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.451182 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.451108    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:38.593538 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:38.593513    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"]
Jul 24 18:01:38.596284 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:01:38.596254    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d49a6cd_cdea_44d4_abb0_d58bb9ef8da9.slice/crio-3cbd6cd14ebb1e5e0569157ce7f08d6be372c6235c4aeed19171bd51e31a60ce WatchSource:0}: Error finding container 3cbd6cd14ebb1e5e0569157ce7f08d6be372c6235c4aeed19171bd51e31a60ce: Status 404 returned error can't find the container with id 3cbd6cd14ebb1e5e0569157ce7f08d6be372c6235c4aeed19171bd51e31a60ce
Jul 24 18:01:39.319751 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:39.319715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5" event={"ID":"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9","Type":"ContainerStarted","Data":"f7911484178a25962987ff434c22ce78d1064081bc808153a7dd735389f31bdd"}
Jul 24 18:01:39.320154 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:39.319766    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5" event={"ID":"2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9","Type":"ContainerStarted","Data":"3cbd6cd14ebb1e5e0569157ce7f08d6be372c6235c4aeed19171bd51e31a60ce"}
Jul 24 18:01:39.320154 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:39.319826    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:39.343355 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:39.343306    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5" podStartSLOduration=1.343291782 podStartE2EDuration="1.343291782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:01:39.341204388 +0000 UTC m=+1290.458131728" watchObservedRunningTime="2026-07-24 18:01:39.343291782 +0000 UTC m=+1290.460219121"
Jul 24 18:01:50.327481 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:50.327450    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vdmx5"
Jul 24 18:01:58.952635 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:01:58.952603    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:02:09.359716 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:02:09.359683    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:02:18.455445 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:02:18.455411    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:02:29.762150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:02:29.762063    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:02:39.245017 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:02:39.244979    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:02:50.050776 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:02:50.050739    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:03:00.552932 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:03:00.552896    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:03:10.846249 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:03:10.846211    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:03:21.547353 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:03:21.547313    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:03:29.645507 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:03:29.645473    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:03:38.460432 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:03:38.460398    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:03:47.958080 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:03:47.958042    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:04:06.159030 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:06.158950    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:04:15.671264 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:15.671222    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:04:18.626964 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.626920    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"]
Jul 24 18:04:18.630438 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.630416    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.632930 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.632913    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6cf28b90-kserve-self-signed-certs\""
Jul 24 18:04:18.647691 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.647669    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"]
Jul 24 18:04:18.709899 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.709866    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc34071-6d23-432a-bac9-7d77042e246b-tls-certs\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.709899 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.709900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbv4g\" (UniqueName: \"kubernetes.io/projected/cdc34071-6d23-432a-bac9-7d77042e246b-kube-api-access-pbv4g\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.710148 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.709928    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-model-cache\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.710148 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.710010    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-tmp-dir\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.710148 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.710050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-dshm\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.710148 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.710113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-home\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.810860 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.810816    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc34071-6d23-432a-bac9-7d77042e246b-tls-certs\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.810860 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.810863    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbv4g\" (UniqueName: \"kubernetes.io/projected/cdc34071-6d23-432a-bac9-7d77042e246b-kube-api-access-pbv4g\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.811121 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.810908    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-model-cache\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.811121 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.810965    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-tmp-dir\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.811121 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.811014    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-dshm\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.811121 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.811072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-home\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.811483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.811454    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-model-cache\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.811661 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.811533    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-tmp-dir\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.811661 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.811553    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-home\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.813486 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.813449    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-dshm\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.813600 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.813578    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc34071-6d23-432a-bac9-7d77042e246b-tls-certs\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.819643 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.819622    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbv4g\" (UniqueName: \"kubernetes.io/projected/cdc34071-6d23-432a-bac9-7d77042e246b-kube-api-access-pbv4g\") pod \"e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") " pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:18.939038 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:18.938951    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:19.081698 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:19.081661    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"]
Jul 24 18:04:19.083444 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:04:19.083419    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc34071_6d23_432a_bac9_7d77042e246b.slice/crio-3aa03f051831fb21892b5588ea4ddc6441aef4db2d72c767b3b7460f4ff43484 WatchSource:0}: Error finding container 3aa03f051831fb21892b5588ea4ddc6441aef4db2d72c767b3b7460f4ff43484: Status 404 returned error can't find the container with id 3aa03f051831fb21892b5588ea4ddc6441aef4db2d72c767b3b7460f4ff43484
Jul 24 18:04:19.096895 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:19.096869    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7" event={"ID":"cdc34071-6d23-432a-bac9-7d77042e246b","Type":"ContainerStarted","Data":"3aa03f051831fb21892b5588ea4ddc6441aef4db2d72c767b3b7460f4ff43484"}
Jul 24 18:04:20.103207 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:20.103171    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7" event={"ID":"cdc34071-6d23-432a-bac9-7d77042e246b","Type":"ContainerStarted","Data":"528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0"}
Jul 24 18:04:20.122597 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:20.122546    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7" podStartSLOduration=2.122529833 podStartE2EDuration="2.122529833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:04:20.120074422 +0000 UTC m=+1451.237001763" watchObservedRunningTime="2026-07-24 18:04:20.122529833 +0000 UTC m=+1451.239457173"
Jul 24 18:04:23.846037 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:23.846001    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:04:28.940027 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:28.939941    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:28.940468 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:28.940212    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:28.952423 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:28.952387    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:29.155726 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:29.155687    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:04:36.099347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.099311    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"]
Jul 24 18:04:36.103826 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.103804    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.106151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.106133    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6cf28b90-kserve-self-signed-certs\""
Jul 24 18:04:36.112804 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.112780    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"]
Jul 24 18:04:36.178755 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.178706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpnz\" (UniqueName: \"kubernetes.io/projected/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-kube-api-access-2zpnz\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.178755 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.178754    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-home\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.179018 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.178861    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tls-certs\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.179018 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.178964    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-model-cache\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.179018 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.178986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tmp-dir\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.179131 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.179037    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-dshm\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.279710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.279668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-dshm\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.279880 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.279719    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpnz\" (UniqueName: \"kubernetes.io/projected/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-kube-api-access-2zpnz\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.279880 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.279748    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-home\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.279880 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.279791    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tls-certs\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.279880 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.279862    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-model-cache\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.280094 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.279977    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tmp-dir\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.280181 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.280161    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-home\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.280241 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.280225    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-model-cache\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.280314 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.280297    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tmp-dir\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.282148 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.282123    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-dshm\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.282563 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.282542    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tls-certs\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.287845 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.287818    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpnz\" (UniqueName: \"kubernetes.io/projected/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-kube-api-access-2zpnz\") pod \"e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") " pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.415189 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.415102    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:36.552479 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:36.552455    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"]
Jul 24 18:04:36.554929 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:04:36.554895    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c577b6f_25d2_46ca_a665_dfc2132b7b0f.slice/crio-de4a4f66738953dad8e63afd5bac510142d8b03c434631eb84d722541f9f1ad5 WatchSource:0}: Error finding container de4a4f66738953dad8e63afd5bac510142d8b03c434631eb84d722541f9f1ad5: Status 404 returned error can't find the container with id de4a4f66738953dad8e63afd5bac510142d8b03c434631eb84d722541f9f1ad5
Jul 24 18:04:37.186094 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:37.186038    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw" event={"ID":"4c577b6f-25d2-46ca-a665-dfc2132b7b0f","Type":"ContainerStarted","Data":"ddd9ce6457e0e959cc4ddef495f0136da4233dc018c8db1d8a059f65178beab1"}
Jul 24 18:04:37.186094 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:37.186097    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw" event={"ID":"4c577b6f-25d2-46ca-a665-dfc2132b7b0f","Type":"ContainerStarted","Data":"de4a4f66738953dad8e63afd5bac510142d8b03c434631eb84d722541f9f1ad5"}
Jul 24 18:04:37.204742 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:37.204689    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw" podStartSLOduration=1.204670064 podStartE2EDuration="1.204670064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:04:37.201638449 +0000 UTC m=+1468.318565789" watchObservedRunningTime="2026-07-24 18:04:37.204670064 +0000 UTC m=+1468.321597421"
Jul 24 18:04:39.951696 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:39.951656    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:04:46.415780 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:46.415742    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:46.416252 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:46.415958    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:46.428277 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:46.428247    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:04:47.244923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:04:47.244893    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:05:05.756452 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:05.756417    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"]
Jul 24 18:05:07.162463 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:07.162430    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"]
Jul 24 18:05:09.315987 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:09.315961    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:05:09.319279 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:09.319257    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:05:09.645604 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:09.645510    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:05:20.776791 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:20.776724    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7" podUID="cdc34071-6d23-432a-bac9-7d77042e246b" containerName="main" containerID="cri-o://528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0" gracePeriod=45
Jul 24 18:05:21.025610 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.025585    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:05:21.114942 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.114903    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-tmp-dir\") pod \"cdc34071-6d23-432a-bac9-7d77042e246b\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") "
Jul 24 18:05:21.115150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.114970    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc34071-6d23-432a-bac9-7d77042e246b-tls-certs\") pod \"cdc34071-6d23-432a-bac9-7d77042e246b\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") "
Jul 24 18:05:21.115150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115002    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-dshm\") pod \"cdc34071-6d23-432a-bac9-7d77042e246b\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") "
Jul 24 18:05:21.115150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115049    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-model-cache\") pod \"cdc34071-6d23-432a-bac9-7d77042e246b\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") "
Jul 24 18:05:21.115150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115073    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbv4g\" (UniqueName: \"kubernetes.io/projected/cdc34071-6d23-432a-bac9-7d77042e246b-kube-api-access-pbv4g\") pod \"cdc34071-6d23-432a-bac9-7d77042e246b\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") "
Jul 24 18:05:21.115150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115103    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-home\") pod \"cdc34071-6d23-432a-bac9-7d77042e246b\" (UID: \"cdc34071-6d23-432a-bac9-7d77042e246b\") "
Jul 24 18:05:21.115458 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115190    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdc34071-6d23-432a-bac9-7d77042e246b" (UID: "cdc34071-6d23-432a-bac9-7d77042e246b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:21.115458 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115300    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdc34071-6d23-432a-bac9-7d77042e246b" (UID: "cdc34071-6d23-432a-bac9-7d77042e246b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:21.115558 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115471    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-home" (OuterVolumeSpecName: "home") pod "cdc34071-6d23-432a-bac9-7d77042e246b" (UID: "cdc34071-6d23-432a-bac9-7d77042e246b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:21.115605 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115578    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:21.115605 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115599    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:21.115674 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.115612    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:21.117414 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.117358    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-dshm" (OuterVolumeSpecName: "dshm") pod "cdc34071-6d23-432a-bac9-7d77042e246b" (UID: "cdc34071-6d23-432a-bac9-7d77042e246b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:21.117534 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.117451    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc34071-6d23-432a-bac9-7d77042e246b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdc34071-6d23-432a-bac9-7d77042e246b" (UID: "cdc34071-6d23-432a-bac9-7d77042e246b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:05:21.117534 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.117467    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc34071-6d23-432a-bac9-7d77042e246b-kube-api-access-pbv4g" (OuterVolumeSpecName: "kube-api-access-pbv4g") pod "cdc34071-6d23-432a-bac9-7d77042e246b" (UID: "cdc34071-6d23-432a-bac9-7d77042e246b"). InnerVolumeSpecName "kube-api-access-pbv4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:05:21.216599 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.216563    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbv4g\" (UniqueName: \"kubernetes.io/projected/cdc34071-6d23-432a-bac9-7d77042e246b-kube-api-access-pbv4g\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:21.216864 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.216836    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc34071-6d23-432a-bac9-7d77042e246b-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:21.216864 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.216863    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc34071-6d23-432a-bac9-7d77042e246b-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:21.397642 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.397547    2551 generic.go:356] "Generic (PLEG): container finished" podID="cdc34071-6d23-432a-bac9-7d77042e246b" containerID="528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0" exitCode=0
Jul 24 18:05:21.397642 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.397602    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"
Jul 24 18:05:21.397642 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.397629    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7" event={"ID":"cdc34071-6d23-432a-bac9-7d77042e246b","Type":"ContainerDied","Data":"528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0"}
Jul 24 18:05:21.397935 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.397681    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7" event={"ID":"cdc34071-6d23-432a-bac9-7d77042e246b","Type":"ContainerDied","Data":"3aa03f051831fb21892b5588ea4ddc6441aef4db2d72c767b3b7460f4ff43484"}
Jul 24 18:05:21.397935 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.397729    2551 scope.go:117] "RemoveContainer" containerID="528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0"
Jul 24 18:05:21.409587 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.409566    2551 scope.go:117] "RemoveContainer" containerID="528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0"
Jul 24 18:05:21.409917 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:05:21.409891    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0\": container with ID starting with 528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0 not found: ID does not exist" containerID="528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0"
Jul 24 18:05:21.410018 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.409920    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0"} err="failed to get container status \"528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0\": rpc error: code = NotFound desc = could not find container \"528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0\": container with ID starting with 528998087e9a22b96e5b0ce4c0ee6206339df4edec84d93c8d5afdb31f266fd0 not found: ID does not exist"
Jul 24 18:05:21.414606 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.414580    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"]
Jul 24 18:05:21.420544 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:21.420521    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6cf28b90-kserve-74897664c5-xftn7"]
Jul 24 18:05:22.182406 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.182347    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw" podUID="4c577b6f-25d2-46ca-a665-dfc2132b7b0f" containerName="main" containerID="cri-o://ddd9ce6457e0e959cc4ddef495f0136da4233dc018c8db1d8a059f65178beab1" gracePeriod=45
Jul 24 18:05:22.404124 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.404095    2551 generic.go:356] "Generic (PLEG): container finished" podID="4c577b6f-25d2-46ca-a665-dfc2132b7b0f" containerID="ddd9ce6457e0e959cc4ddef495f0136da4233dc018c8db1d8a059f65178beab1" exitCode=0
Jul 24 18:05:22.404289 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.404169    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw" event={"ID":"4c577b6f-25d2-46ca-a665-dfc2132b7b0f","Type":"ContainerDied","Data":"ddd9ce6457e0e959cc4ddef495f0136da4233dc018c8db1d8a059f65178beab1"}
Jul 24 18:05:22.432746 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.432691    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:05:22.527106 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527074    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zpnz\" (UniqueName: \"kubernetes.io/projected/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-kube-api-access-2zpnz\") pod \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") "
Jul 24 18:05:22.527285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527158    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-model-cache\") pod \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") "
Jul 24 18:05:22.527285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527230    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tls-certs\") pod \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") "
Jul 24 18:05:22.527285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527258    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tmp-dir\") pod \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") "
Jul 24 18:05:22.527285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527282    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-dshm\") pod \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") "
Jul 24 18:05:22.527575 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527307    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-home\") pod \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\" (UID: \"4c577b6f-25d2-46ca-a665-dfc2132b7b0f\") "
Jul 24 18:05:22.527575 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527458    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c577b6f-25d2-46ca-a665-dfc2132b7b0f" (UID: "4c577b6f-25d2-46ca-a665-dfc2132b7b0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:22.527677 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527584    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-home" (OuterVolumeSpecName: "home") pod "4c577b6f-25d2-46ca-a665-dfc2132b7b0f" (UID: "4c577b6f-25d2-46ca-a665-dfc2132b7b0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:22.527677 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527594    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c577b6f-25d2-46ca-a665-dfc2132b7b0f" (UID: "4c577b6f-25d2-46ca-a665-dfc2132b7b0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:22.527781 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527719    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:22.527781 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527734    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:22.527781 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.527743    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:22.529448 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.529423    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c577b6f-25d2-46ca-a665-dfc2132b7b0f" (UID: "4c577b6f-25d2-46ca-a665-dfc2132b7b0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:05:22.529448 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.529429    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-kube-api-access-2zpnz" (OuterVolumeSpecName: "kube-api-access-2zpnz") pod "4c577b6f-25d2-46ca-a665-dfc2132b7b0f" (UID: "4c577b6f-25d2-46ca-a665-dfc2132b7b0f"). InnerVolumeSpecName "kube-api-access-2zpnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:05:22.529448 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.529437    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-dshm" (OuterVolumeSpecName: "dshm") pod "4c577b6f-25d2-46ca-a665-dfc2132b7b0f" (UID: "4c577b6f-25d2-46ca-a665-dfc2132b7b0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:05:22.628559 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.628524    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:22.628559 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.628554    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:22.628559 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.628563    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpnz\" (UniqueName: \"kubernetes.io/projected/4c577b6f-25d2-46ca-a665-dfc2132b7b0f-kube-api-access-2zpnz\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:05:22.646596 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:22.646564    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:05:23.300585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:23.300550    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc34071-6d23-432a-bac9-7d77042e246b" path="/var/lib/kubelet/pods/cdc34071-6d23-432a-bac9-7d77042e246b/volumes"
Jul 24 18:05:23.410626 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:23.410601    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"
Jul 24 18:05:23.410816 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:23.410598    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw" event={"ID":"4c577b6f-25d2-46ca-a665-dfc2132b7b0f","Type":"ContainerDied","Data":"de4a4f66738953dad8e63afd5bac510142d8b03c434631eb84d722541f9f1ad5"}
Jul 24 18:05:23.410816 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:23.410716    2551 scope.go:117] "RemoveContainer" containerID="ddd9ce6457e0e959cc4ddef495f0136da4233dc018c8db1d8a059f65178beab1"
Jul 24 18:05:23.427286 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:23.427258    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"]
Jul 24 18:05:23.430876 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:23.430853    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6cf28b90-kserve-6cc48bd6f5-ms5gw"]
Jul 24 18:05:25.301016 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:25.300980    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c577b6f-25d2-46ca-a665-dfc2132b7b0f" path="/var/lib/kubelet/pods/4c577b6f-25d2-46ca-a665-dfc2132b7b0f/volumes"
Jul 24 18:05:29.750422 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:29.750325    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:05:45.652685 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:45.652645    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:05:55.848398 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:05:55.848348    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:06:04.551825 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:06:04.551788    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:06:16.366033 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:06:16.365997    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:06:25.718385 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:06:25.718349    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:06:34.945337 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:06:34.945303    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:06:48.150081 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:06:48.150047    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:06:57.754767 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:06:57.754735    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:07:07.441715 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:07:07.441632    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:07:18.145978 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:07:18.145925    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:07:28.652827 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:07:28.652792    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:07:39.059089 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:07:39.059052    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:07:51.349742 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:07:51.349706    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:08:03.546006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:03.545967    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:08:11.845595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:11.845556    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:08:21.447590 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:21.447552    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:08:34.747795 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:34.747698    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:08:43.943957 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:43.943919    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:08:52.231119 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.231083    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"]
Jul 24 18:08:52.231501 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.231489    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc34071-6d23-432a-bac9-7d77042e246b" containerName="main"
Jul 24 18:08:52.231501 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.231502    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc34071-6d23-432a-bac9-7d77042e246b" containerName="main"
Jul 24 18:08:52.231574 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.231511    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c577b6f-25d2-46ca-a665-dfc2132b7b0f" containerName="main"
Jul 24 18:08:52.231574 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.231516    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c577b6f-25d2-46ca-a665-dfc2132b7b0f" containerName="main"
Jul 24 18:08:52.231636 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.231585    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc34071-6d23-432a-bac9-7d77042e246b" containerName="main"
Jul 24 18:08:52.231636 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.231596    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c577b6f-25d2-46ca-a665-dfc2132b7b0f" containerName="main"
Jul 24 18:08:52.234977 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.234953    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.237293 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.237269    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3e8e6244-serving-cert\""
Jul 24 18:08:52.237293 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.237284    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:08:52.237512 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.237327    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:08:52.238333 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.238306    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:08:52.245119 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.245091    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"]
Jul 24 18:08:52.362011 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.361967    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgd9g\" (UniqueName: \"kubernetes.io/projected/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-kube-api-access-qgd9g\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.362200 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.362097    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-openshift-service-ca\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.362200 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.362171    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-maas-api-tls\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.463766 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.463713    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-openshift-service-ca\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.463939 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.463830    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-maas-api-tls\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.464011 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.463964    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgd9g\" (UniqueName: \"kubernetes.io/projected/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-kube-api-access-qgd9g\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.464466 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.464440    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-openshift-service-ca\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.466448 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.466427    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-maas-api-tls\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.473574 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.473555    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgd9g\" (UniqueName: \"kubernetes.io/projected/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-kube-api-access-qgd9g\") pod \"maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.546692 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.546609    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:52.715398 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.715341    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"]
Jul 24 18:08:52.718194 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:08:52.718135    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799cc3c3_d45e_47b5_83ce_8b6afe3968d5.slice/crio-355ecf82a1ea3e9c19fecff5b29d775975d8acde1ecefa0d68065bdc28f6fb74 WatchSource:0}: Error finding container 355ecf82a1ea3e9c19fecff5b29d775975d8acde1ecefa0d68065bdc28f6fb74: Status 404 returned error can't find the container with id 355ecf82a1ea3e9c19fecff5b29d775975d8acde1ecefa0d68065bdc28f6fb74
Jul 24 18:08:52.719612 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:52.719591    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:08:53.409628 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:53.409591    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q" event={"ID":"799cc3c3-d45e-47b5-83ce-8b6afe3968d5","Type":"ContainerStarted","Data":"355ecf82a1ea3e9c19fecff5b29d775975d8acde1ecefa0d68065bdc28f6fb74"}
Jul 24 18:08:55.427261 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:55.427226    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q" event={"ID":"799cc3c3-d45e-47b5-83ce-8b6afe3968d5","Type":"ContainerStarted","Data":"65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154"}
Jul 24 18:08:55.427690 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:55.427354    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:08:55.445532 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:08:55.445475    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q" podStartSLOduration=1.385786885 podStartE2EDuration="3.445454706s" totalImagesPullingTime="2.059667821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:08:55.444983749 +0000 UTC m=+1726.561911095" watchObservedRunningTime="2026-07-24 18:08:55.445454706 +0000 UTC m=+1726.562382049"
Jul 24 18:09:00.050796 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:00.050760    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:09:01.438207 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:01.438162    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:09:28.185808 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.185764    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"]
Jul 24 18:09:28.186329 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.186049    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q" podUID="799cc3c3-d45e-47b5-83ce-8b6afe3968d5" containerName="maas-api" containerID="cri-o://65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154" gracePeriod=30
Jul 24 18:09:28.446787 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.446725    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:09:28.586258 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.586220    2551 generic.go:356] "Generic (PLEG): container finished" podID="799cc3c3-d45e-47b5-83ce-8b6afe3968d5" containerID="65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154" exitCode=0
Jul 24 18:09:28.586469 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.586274    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"
Jul 24 18:09:28.586469 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.586296    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q" event={"ID":"799cc3c3-d45e-47b5-83ce-8b6afe3968d5","Type":"ContainerDied","Data":"65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154"}
Jul 24 18:09:28.586469 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.586350    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q" event={"ID":"799cc3c3-d45e-47b5-83ce-8b6afe3968d5","Type":"ContainerDied","Data":"355ecf82a1ea3e9c19fecff5b29d775975d8acde1ecefa0d68065bdc28f6fb74"}
Jul 24 18:09:28.586469 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.586402    2551 scope.go:117] "RemoveContainer" containerID="65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154"
Jul 24 18:09:28.597263 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.597241    2551 scope.go:117] "RemoveContainer" containerID="65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154"
Jul 24 18:09:28.597832 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:09:28.597811    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154\": container with ID starting with 65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154 not found: ID does not exist" containerID="65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154"
Jul 24 18:09:28.597928 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.597846    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154"} err="failed to get container status \"65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154\": rpc error: code = NotFound desc = could not find container \"65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154\": container with ID starting with 65ecc7b7d2248c2dabdfbcab8945936339484fc1d11df5a2645f1a3cbabb8154 not found: ID does not exist"
Jul 24 18:09:28.604118 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.604098    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-maas-api-tls\") pod \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") "
Jul 24 18:09:28.604206 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.604152    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgd9g\" (UniqueName: \"kubernetes.io/projected/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-kube-api-access-qgd9g\") pod \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") "
Jul 24 18:09:28.604206 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.604197    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-openshift-service-ca\") pod \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\" (UID: \"799cc3c3-d45e-47b5-83ce-8b6afe3968d5\") "
Jul 24 18:09:28.604645 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.604616    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "799cc3c3-d45e-47b5-83ce-8b6afe3968d5" (UID: "799cc3c3-d45e-47b5-83ce-8b6afe3968d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:09:28.606412 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.606368    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "799cc3c3-d45e-47b5-83ce-8b6afe3968d5" (UID: "799cc3c3-d45e-47b5-83ce-8b6afe3968d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:09:28.606502 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.606455    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-kube-api-access-qgd9g" (OuterVolumeSpecName: "kube-api-access-qgd9g") pod "799cc3c3-d45e-47b5-83ce-8b6afe3968d5" (UID: "799cc3c3-d45e-47b5-83ce-8b6afe3968d5"). InnerVolumeSpecName "kube-api-access-qgd9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:09:28.705442 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.705346    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:09:28.705442 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.705388    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgd9g\" (UniqueName: \"kubernetes.io/projected/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-kube-api-access-qgd9g\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:09:28.705442 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.705400    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/799cc3c3-d45e-47b5-83ce-8b6afe3968d5-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:09:28.910170 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.910133    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"]
Jul 24 18:09:28.914321 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:28.914297    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e8e6244-db6f7dd5b-fh77q"]
Jul 24 18:09:29.300738 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:29.300693    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="799cc3c3-d45e-47b5-83ce-8b6afe3968d5" path="/var/lib/kubelet/pods/799cc3c3-d45e-47b5-83ce-8b6afe3968d5/volumes"
Jul 24 18:09:31.153494 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:31.153458    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:09:36.386573 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.386532    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"]
Jul 24 18:09:36.387150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.387083    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="799cc3c3-d45e-47b5-83ce-8b6afe3968d5" containerName="maas-api"
Jul 24 18:09:36.387150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.387100    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="799cc3c3-d45e-47b5-83ce-8b6afe3968d5" containerName="maas-api"
Jul 24 18:09:36.387269 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.387184    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="799cc3c3-d45e-47b5-83ce-8b6afe3968d5" containerName="maas-api"
Jul 24 18:09:36.391915 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.391887    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.394328 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.394290    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8be87499-serving-cert\""
Jul 24 18:09:36.394554 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.394440    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:09:36.394554 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.394299    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:09:36.395083 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.395053    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:09:36.400801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.400774    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"]
Jul 24 18:09:36.580946 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.580903    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.581130 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.580988    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spzjb\" (UniqueName: \"kubernetes.io/projected/8c60958b-123c-4c5e-88c1-c4390a179583-kube-api-access-spzjb\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.581130 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.581025    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c60958b-123c-4c5e-88c1-c4390a179583-openshift-service-ca\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.681752 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.681644    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.681752 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.681745    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzjb\" (UniqueName: \"kubernetes.io/projected/8c60958b-123c-4c5e-88c1-c4390a179583-kube-api-access-spzjb\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.681969 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.681779    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c60958b-123c-4c5e-88c1-c4390a179583-openshift-service-ca\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.681969 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:09:36.681805    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-serving-cert: secret "maas-api-e2e-migrate-8be87499-serving-cert" not found
Jul 24 18:09:36.681969 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:09:36.681876    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls podName:8c60958b-123c-4c5e-88c1-c4390a179583 nodeName:}" failed. No retries permitted until 2026-07-24 18:09:37.181858334 +0000 UTC m=+1768.298785652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls") pod "maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr" (UID: "8c60958b-123c-4c5e-88c1-c4390a179583") : secret "maas-api-e2e-migrate-8be87499-serving-cert" not found
Jul 24 18:09:36.682358 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.682339    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c60958b-123c-4c5e-88c1-c4390a179583-openshift-service-ca\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:36.692804 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:36.692770    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzjb\" (UniqueName: \"kubernetes.io/projected/8c60958b-123c-4c5e-88c1-c4390a179583-kube-api-access-spzjb\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:37.095774 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.095738    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"]
Jul 24 18:09:37.099519 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.099497    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.101752 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.101728    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8be87499\""
Jul 24 18:09:37.101854 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.101729    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8be87499-dockercfg-gwdx5\""
Jul 24 18:09:37.106978 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.106951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"]
Jul 24 18:09:37.188156 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.188126    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:37.190824 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.190795    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls\") pod \"maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:37.288941 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.288897    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58m2\" (UniqueName: \"kubernetes.io/projected/2ee70242-a885-4c1b-9e78-dce9fc938259-kube-api-access-l58m2\") pod \"payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.289142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.288986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee70242-a885-4c1b-9e78-dce9fc938259-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.306255 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.306224    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:37.389749 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.389675    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58m2\" (UniqueName: \"kubernetes.io/projected/2ee70242-a885-4c1b-9e78-dce9fc938259-kube-api-access-l58m2\") pod \"payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.390335 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.390312    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee70242-a885-4c1b-9e78-dce9fc938259-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.391411 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.391258    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee70242-a885-4c1b-9e78-dce9fc938259-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.399146 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.399106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58m2\" (UniqueName: \"kubernetes.io/projected/2ee70242-a885-4c1b-9e78-dce9fc938259-kube-api-access-l58m2\") pod \"payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.413130 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.413057    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:37.448756 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.448731    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"]
Jul 24 18:09:37.563600 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.563515    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"]
Jul 24 18:09:37.565653 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:09:37.565624    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee70242_a885_4c1b_9e78_dce9fc938259.slice/crio-e1235f3c5928841317a48d00aed6049ba962f18421342cb136e107897335b728 WatchSource:0}: Error finding container e1235f3c5928841317a48d00aed6049ba962f18421342cb136e107897335b728: Status 404 returned error can't find the container with id e1235f3c5928841317a48d00aed6049ba962f18421342cb136e107897335b728
Jul 24 18:09:37.634820 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.634721    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j" event={"ID":"2ee70242-a885-4c1b-9e78-dce9fc938259","Type":"ContainerStarted","Data":"e1235f3c5928841317a48d00aed6049ba962f18421342cb136e107897335b728"}
Jul 24 18:09:37.635873 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:37.635842    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr" event={"ID":"8c60958b-123c-4c5e-88c1-c4390a179583","Type":"ContainerStarted","Data":"d2923c737bf432cb5fa29fc380e247a0c5e76ec083d38b3fecc1a1c727176cc4"}
Jul 24 18:09:38.644634 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:38.644582    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr" event={"ID":"8c60958b-123c-4c5e-88c1-c4390a179583","Type":"ContainerStarted","Data":"ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779"}
Jul 24 18:09:38.645161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:38.644807    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:38.662558 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:38.662489    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr" podStartSLOduration=2.317939565 podStartE2EDuration="2.662465613s" totalImagesPullingTime="344.526048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:38.661645614 +0000 UTC m=+1769.778572948" watchObservedRunningTime="2026-07-24 18:09:38.662465613 +0000 UTC m=+1769.779392958"
Jul 24 18:09:39.651915 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:39.651878    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j" event={"ID":"2ee70242-a885-4c1b-9e78-dce9fc938259","Type":"ContainerStarted","Data":"b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be"}
Jul 24 18:09:39.652411 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:39.652261    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:09:39.669332 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:39.669287    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j" podStartSLOduration=1.128520791 podStartE2EDuration="2.669272964s" totalImagesPullingTime="1.540752173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:39.668069943 +0000 UTC m=+1770.784997284" watchObservedRunningTime="2026-07-24 18:09:39.669272964 +0000 UTC m=+1770.786200304"
Jul 24 18:09:42.548852 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:42.548812    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:09:44.657420 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:44.657361    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:09:50.659248 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:09:50.659217    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:10:09.345982 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:09.345956    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:10:09.350883 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:09.350864    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:10:11.826599 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:11.826566    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"]
Jul 24 18:10:11.827013 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:11.826809    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr" podUID="8c60958b-123c-4c5e-88c1-c4390a179583" containerName="maas-api" containerID="cri-o://ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779" gracePeriod=30
Jul 24 18:10:11.885127 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:11.885095    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"]
Jul 24 18:10:11.885325 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:11.885301    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j" podUID="2ee70242-a885-4c1b-9e78-dce9fc938259" containerName="payload-pre-processing" containerID="cri-o://b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be" gracePeriod=30
Jul 24 18:10:12.165520 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.165494    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:10:12.169658 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.169627    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:10:12.318352 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.318313    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c60958b-123c-4c5e-88c1-c4390a179583-openshift-service-ca\") pod \"8c60958b-123c-4c5e-88c1-c4390a179583\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") "
Jul 24 18:10:12.318352 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.318355    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l58m2\" (UniqueName: \"kubernetes.io/projected/2ee70242-a885-4c1b-9e78-dce9fc938259-kube-api-access-l58m2\") pod \"2ee70242-a885-4c1b-9e78-dce9fc938259\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") "
Jul 24 18:10:12.318603 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.318399    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee70242-a885-4c1b-9e78-dce9fc938259-plugins-config-volume\") pod \"2ee70242-a885-4c1b-9e78-dce9fc938259\" (UID: \"2ee70242-a885-4c1b-9e78-dce9fc938259\") "
Jul 24 18:10:12.318603 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.318480    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls\") pod \"8c60958b-123c-4c5e-88c1-c4390a179583\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") "
Jul 24 18:10:12.318603 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.318516    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spzjb\" (UniqueName: \"kubernetes.io/projected/8c60958b-123c-4c5e-88c1-c4390a179583-kube-api-access-spzjb\") pod \"8c60958b-123c-4c5e-88c1-c4390a179583\" (UID: \"8c60958b-123c-4c5e-88c1-c4390a179583\") "
Jul 24 18:10:12.318788 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.318757    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c60958b-123c-4c5e-88c1-c4390a179583-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c60958b-123c-4c5e-88c1-c4390a179583" (UID: "8c60958b-123c-4c5e-88c1-c4390a179583"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:12.318849 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.318808    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee70242-a885-4c1b-9e78-dce9fc938259-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ee70242-a885-4c1b-9e78-dce9fc938259" (UID: "2ee70242-a885-4c1b-9e78-dce9fc938259"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:12.320701 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.320671    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c60958b-123c-4c5e-88c1-c4390a179583-kube-api-access-spzjb" (OuterVolumeSpecName: "kube-api-access-spzjb") pod "8c60958b-123c-4c5e-88c1-c4390a179583" (UID: "8c60958b-123c-4c5e-88c1-c4390a179583"). InnerVolumeSpecName "kube-api-access-spzjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:12.320807 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.320718    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c60958b-123c-4c5e-88c1-c4390a179583" (UID: "8c60958b-123c-4c5e-88c1-c4390a179583"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:10:12.320905 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.320891    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee70242-a885-4c1b-9e78-dce9fc938259-kube-api-access-l58m2" (OuterVolumeSpecName: "kube-api-access-l58m2") pod "2ee70242-a885-4c1b-9e78-dce9fc938259" (UID: "2ee70242-a885-4c1b-9e78-dce9fc938259"). InnerVolumeSpecName "kube-api-access-l58m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:12.419830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.419747    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spzjb\" (UniqueName: \"kubernetes.io/projected/8c60958b-123c-4c5e-88c1-c4390a179583-kube-api-access-spzjb\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:12.419830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.419777    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c60958b-123c-4c5e-88c1-c4390a179583-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:12.419830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.419788    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58m2\" (UniqueName: \"kubernetes.io/projected/2ee70242-a885-4c1b-9e78-dce9fc938259-kube-api-access-l58m2\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:12.419830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.419798    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee70242-a885-4c1b-9e78-dce9fc938259-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:12.419830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.419807    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c60958b-123c-4c5e-88c1-c4390a179583-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:12.813925 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.813886    2551 generic.go:356] "Generic (PLEG): container finished" podID="2ee70242-a885-4c1b-9e78-dce9fc938259" containerID="b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be" exitCode=0
Jul 24 18:10:12.814163 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.813935    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"
Jul 24 18:10:12.814163 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.813973    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j" event={"ID":"2ee70242-a885-4c1b-9e78-dce9fc938259","Type":"ContainerDied","Data":"b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be"}
Jul 24 18:10:12.814163 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.814027    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j" event={"ID":"2ee70242-a885-4c1b-9e78-dce9fc938259","Type":"ContainerDied","Data":"e1235f3c5928841317a48d00aed6049ba962f18421342cb136e107897335b728"}
Jul 24 18:10:12.814163 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.814057    2551 scope.go:117] "RemoveContainer" containerID="b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be"
Jul 24 18:10:12.815445 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.815422    2551 generic.go:356] "Generic (PLEG): container finished" podID="8c60958b-123c-4c5e-88c1-c4390a179583" containerID="ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779" exitCode=0
Jul 24 18:10:12.815552 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.815456    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr" event={"ID":"8c60958b-123c-4c5e-88c1-c4390a179583","Type":"ContainerDied","Data":"ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779"}
Jul 24 18:10:12.815552 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.815464    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"
Jul 24 18:10:12.815552 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.815494    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr" event={"ID":"8c60958b-123c-4c5e-88c1-c4390a179583","Type":"ContainerDied","Data":"d2923c737bf432cb5fa29fc380e247a0c5e76ec083d38b3fecc1a1c727176cc4"}
Jul 24 18:10:12.823649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.823632    2551 scope.go:117] "RemoveContainer" containerID="b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be"
Jul 24 18:10:12.823946 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:10:12.823923    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be\": container with ID starting with b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be not found: ID does not exist" containerID="b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be"
Jul 24 18:10:12.824032 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.823953    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be"} err="failed to get container status \"b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be\": rpc error: code = NotFound desc = could not find container \"b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be\": container with ID starting with b0e176694dc3e146df51e2a2073fd9c23189b1572a3eb8831354eaadd520f9be not found: ID does not exist"
Jul 24 18:10:12.824032 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.823973    2551 scope.go:117] "RemoveContainer" containerID="ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779"
Jul 24 18:10:12.832571 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.832388    2551 scope.go:117] "RemoveContainer" containerID="ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779"
Jul 24 18:10:12.832819 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:10:12.832687    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779\": container with ID starting with ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779 not found: ID does not exist" containerID="ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779"
Jul 24 18:10:12.832819 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.832712    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779"} err="failed to get container status \"ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779\": rpc error: code = NotFound desc = could not find container \"ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779\": container with ID starting with ba9a000b56baebbf621c2c72e026199b52e549dc1ac978dc3c75bc278575f779 not found: ID does not exist"
Jul 24 18:10:12.842165 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.842111    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"]
Jul 24 18:10:12.849628 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.846676    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8be87499-69d44c4d44-d92gr"]
Jul 24 18:10:12.860487 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.860463    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"]
Jul 24 18:10:12.865504 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:12.865482    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8be87499-5f59bf7ff6-w958j"]
Jul 24 18:10:13.302412 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:13.302354    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee70242-a885-4c1b-9e78-dce9fc938259" path="/var/lib/kubelet/pods/2ee70242-a885-4c1b-9e78-dce9fc938259/volumes"
Jul 24 18:10:13.302953 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:13.302933    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c60958b-123c-4c5e-88c1-c4390a179583" path="/var/lib/kubelet/pods/8c60958b-123c-4c5e-88c1-c4390a179583/volumes"
Jul 24 18:10:14.948728 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:14.948687    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:10:19.475834 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.475797    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"]
Jul 24 18:10:19.476212 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.476184    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c60958b-123c-4c5e-88c1-c4390a179583" containerName="maas-api"
Jul 24 18:10:19.476212 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.476196    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c60958b-123c-4c5e-88c1-c4390a179583" containerName="maas-api"
Jul 24 18:10:19.476212 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.476212    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee70242-a885-4c1b-9e78-dce9fc938259" containerName="payload-pre-processing"
Jul 24 18:10:19.476320 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.476217    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee70242-a885-4c1b-9e78-dce9fc938259" containerName="payload-pre-processing"
Jul 24 18:10:19.476320 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.476283    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c60958b-123c-4c5e-88c1-c4390a179583" containerName="maas-api"
Jul 24 18:10:19.476320 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.476291    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee70242-a885-4c1b-9e78-dce9fc938259" containerName="payload-pre-processing"
Jul 24 18:10:19.479746 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.479725    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.482038 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.482017    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:10:19.482209 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.482187    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:10:19.482883 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.482866    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:10:19.482964 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.482906    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0858c257-serving-cert\""
Jul 24 18:10:19.495731 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.495703    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"]
Jul 24 18:10:19.581976 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.581939    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266cec25-978d-4bb7-8819-e2771d4d16d3-openshift-service-ca\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.581976 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.581979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdd7\" (UniqueName: \"kubernetes.io/projected/266cec25-978d-4bb7-8819-e2771d4d16d3-kube-api-access-9rdd7\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.582191 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.582060    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.683498 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.683462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.683702 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.683548    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266cec25-978d-4bb7-8819-e2771d4d16d3-openshift-service-ca\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.683702 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.683576    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdd7\" (UniqueName: \"kubernetes.io/projected/266cec25-978d-4bb7-8819-e2771d4d16d3-kube-api-access-9rdd7\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.683702 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:10:19.683624    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-serving-cert: secret "maas-api-e2e-ait-0858c257-serving-cert" not found
Jul 24 18:10:19.683880 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:10:19.683712    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls podName:266cec25-978d-4bb7-8819-e2771d4d16d3 nodeName:}" failed. No retries permitted until 2026-07-24 18:10:20.183690244 +0000 UTC m=+1811.300617563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls") pod "maas-api-e2e-ait-0858c257-f67c586c5-rgtn2" (UID: "266cec25-978d-4bb7-8819-e2771d4d16d3") : secret "maas-api-e2e-ait-0858c257-serving-cert" not found
Jul 24 18:10:19.684146 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.684126    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266cec25-978d-4bb7-8819-e2771d4d16d3-openshift-service-ca\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:19.693142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:19.693111    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdd7\" (UniqueName: \"kubernetes.io/projected/266cec25-978d-4bb7-8819-e2771d4d16d3-kube-api-access-9rdd7\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:20.188813 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.188778    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:20.192002 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.191974    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls\") pod \"maas-api-e2e-ait-0858c257-f67c586c5-rgtn2\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:20.266536 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.266504    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"]
Jul 24 18:10:20.269119 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.269098    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.271811 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.271792    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0858c257\""
Jul 24 18:10:20.278510 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.278484    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"]
Jul 24 18:10:20.289511 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.289487    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbsv\" (UniqueName: \"kubernetes.io/projected/54d0d18e-8c44-4f1f-a922-5252486b96b5-kube-api-access-cjbsv\") pod \"payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.289630 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.289570    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d0d18e-8c44-4f1f-a922-5252486b96b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.307102 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.307070    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"]
Jul 24 18:10:20.309422 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.309403    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.311876 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.311856    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0858c257-dockercfg-jb7t8\""
Jul 24 18:10:20.323151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.323129    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"]
Jul 24 18:10:20.390977 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.390942    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbsv\" (UniqueName: \"kubernetes.io/projected/54d0d18e-8c44-4f1f-a922-5252486b96b5-kube-api-access-cjbsv\") pod \"payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.391175 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.390981    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pss7\" (UniqueName: \"kubernetes.io/projected/0f8147c9-0596-419b-88e3-2c3a32d3cc45-kube-api-access-4pss7\") pod \"payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") " pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.391175 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.391008    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8147c9-0596-419b-88e3-2c3a32d3cc45-plugins-config-volume\") pod \"payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") " pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.391175 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.391039    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:20.391175 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.391087    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d0d18e-8c44-4f1f-a922-5252486b96b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.391799 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.391768    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d0d18e-8c44-4f1f-a922-5252486b96b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.399426 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.399404    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbsv\" (UniqueName: \"kubernetes.io/projected/54d0d18e-8c44-4f1f-a922-5252486b96b5-kube-api-access-cjbsv\") pod \"payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.492349 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.492294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pss7\" (UniqueName: \"kubernetes.io/projected/0f8147c9-0596-419b-88e3-2c3a32d3cc45-kube-api-access-4pss7\") pod \"payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") " pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.492349 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.492343    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8147c9-0596-419b-88e3-2c3a32d3cc45-plugins-config-volume\") pod \"payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") " pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.493354 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.493329    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8147c9-0596-419b-88e3-2c3a32d3cc45-plugins-config-volume\") pod \"payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") " pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.503886 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.503861    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pss7\" (UniqueName: \"kubernetes.io/projected/0f8147c9-0596-419b-88e3-2c3a32d3cc45-kube-api-access-4pss7\") pod \"payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") " pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.531439 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.531412    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"]
Jul 24 18:10:20.533798 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:10:20.533771    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266cec25_978d_4bb7_8819_e2771d4d16d3.slice/crio-8a2b8cd92befd9b6aad1a6cf8b4a56e3c84410e1fe5f795bfddb927089d55d4a WatchSource:0}: Error finding container 8a2b8cd92befd9b6aad1a6cf8b4a56e3c84410e1fe5f795bfddb927089d55d4a: Status 404 returned error can't find the container with id 8a2b8cd92befd9b6aad1a6cf8b4a56e3c84410e1fe5f795bfddb927089d55d4a
Jul 24 18:10:20.580988 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.580957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.619472 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.619442    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.732993 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.732950    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"]
Jul 24 18:10:20.737929 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:10:20.737853    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d0d18e_8c44_4f1f_a922_5252486b96b5.slice/crio-a660766889dbe2210a5874443688a948939db091230bc4897db3cd680b644dff WatchSource:0}: Error finding container a660766889dbe2210a5874443688a948939db091230bc4897db3cd680b644dff: Status 404 returned error can't find the container with id a660766889dbe2210a5874443688a948939db091230bc4897db3cd680b644dff
Jul 24 18:10:20.780429 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.780369    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"]
Jul 24 18:10:20.783654 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:10:20.783624    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8147c9_0596_419b_88e3_2c3a32d3cc45.slice/crio-603f699e40c34bea6d16c701d09d5d796e72a9ec9ab2d8a2f32c0dc23de53908 WatchSource:0}: Error finding container 603f699e40c34bea6d16c701d09d5d796e72a9ec9ab2d8a2f32c0dc23de53908: Status 404 returned error can't find the container with id 603f699e40c34bea6d16c701d09d5d796e72a9ec9ab2d8a2f32c0dc23de53908
Jul 24 18:10:20.861066 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.861030    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn" event={"ID":"0f8147c9-0596-419b-88e3-2c3a32d3cc45","Type":"ContainerStarted","Data":"3b4b53991c288db565eef485b14094a28cbf38486812258ee925dd627651211a"}
Jul 24 18:10:20.861277 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.861094    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn" event={"ID":"0f8147c9-0596-419b-88e3-2c3a32d3cc45","Type":"ContainerStarted","Data":"603f699e40c34bea6d16c701d09d5d796e72a9ec9ab2d8a2f32c0dc23de53908"}
Jul 24 18:10:20.861277 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.861137    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:20.862518 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.862494    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2" event={"ID":"266cec25-978d-4bb7-8819-e2771d4d16d3","Type":"ContainerStarted","Data":"8a2b8cd92befd9b6aad1a6cf8b4a56e3c84410e1fe5f795bfddb927089d55d4a"}
Jul 24 18:10:20.864219 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.864185    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz" event={"ID":"54d0d18e-8c44-4f1f-a922-5252486b96b5","Type":"ContainerStarted","Data":"73d685569e45c045fe0b9ffbe17254051d17c426584e2e893bd2026e884df218"}
Jul 24 18:10:20.864319 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.864227    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz" event={"ID":"54d0d18e-8c44-4f1f-a922-5252486b96b5","Type":"ContainerStarted","Data":"a660766889dbe2210a5874443688a948939db091230bc4897db3cd680b644dff"}
Jul 24 18:10:20.864419 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.864319    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:20.878128 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.878082    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn" podStartSLOduration=0.878063964 podStartE2EDuration="878.063964ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:10:20.876369797 +0000 UTC m=+1811.993297140" watchObservedRunningTime="2026-07-24 18:10:20.878063964 +0000 UTC m=+1811.994991302"
Jul 24 18:10:20.893286 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:20.893222    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz" podStartSLOduration=0.89320175 podStartE2EDuration="893.20175ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:10:20.89259408 +0000 UTC m=+1812.009521421" watchObservedRunningTime="2026-07-24 18:10:20.89320175 +0000 UTC m=+1812.010129091"
Jul 24 18:10:21.871084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:21.871036    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2" event={"ID":"266cec25-978d-4bb7-8819-e2771d4d16d3","Type":"ContainerStarted","Data":"c4d9d5bdcf39c544f2415647502c93867a32fad3bb78f4a6949d7821121b40f8"}
Jul 24 18:10:21.871598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:21.871438    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:21.902805 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:21.902745    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2" podStartSLOduration=2.502691043 podStartE2EDuration="2.902727363s" totalImagesPullingTime="400.03632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:10:21.889665589 +0000 UTC m=+1813.006592930" watchObservedRunningTime="2026-07-24 18:10:21.902727363 +0000 UTC m=+1813.019654702"
Jul 24 18:10:26.157793 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:26.157758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:10:28.886303 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:28.886272    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:31.872988 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:31.872958    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:31.873429 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:31.873034    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:55.814992 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:55.814950    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"]
Jul 24 18:10:55.817779 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:55.815448    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2" podUID="266cec25-978d-4bb7-8819-e2771d4d16d3" containerName="maas-api" containerID="cri-o://c4d9d5bdcf39c544f2415647502c93867a32fad3bb78f4a6949d7821121b40f8" gracePeriod=30
Jul 24 18:10:55.887021 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:55.886982    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"]
Jul 24 18:10:55.887551 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:55.887482    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn" podUID="0f8147c9-0596-419b-88e3-2c3a32d3cc45" containerName="payload-processing" containerID="cri-o://3b4b53991c288db565eef485b14094a28cbf38486812258ee925dd627651211a" gracePeriod=30
Jul 24 18:10:55.894435 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:55.894405    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"]
Jul 24 18:10:55.894700 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:55.894650    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz" podUID="54d0d18e-8c44-4f1f-a922-5252486b96b5" containerName="payload-pre-processing" containerID="cri-o://73d685569e45c045fe0b9ffbe17254051d17c426584e2e893bd2026e884df218" gracePeriod=30
Jul 24 18:10:56.047461 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.047420    2551 generic.go:356] "Generic (PLEG): container finished" podID="54d0d18e-8c44-4f1f-a922-5252486b96b5" containerID="73d685569e45c045fe0b9ffbe17254051d17c426584e2e893bd2026e884df218" exitCode=0
Jul 24 18:10:56.047636 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.047464    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz" event={"ID":"54d0d18e-8c44-4f1f-a922-5252486b96b5","Type":"ContainerDied","Data":"73d685569e45c045fe0b9ffbe17254051d17c426584e2e893bd2026e884df218"}
Jul 24 18:10:56.049450 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.049400    2551 generic.go:356] "Generic (PLEG): container finished" podID="0f8147c9-0596-419b-88e3-2c3a32d3cc45" containerID="3b4b53991c288db565eef485b14094a28cbf38486812258ee925dd627651211a" exitCode=0
Jul 24 18:10:56.049612 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.049493    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn" event={"ID":"0f8147c9-0596-419b-88e3-2c3a32d3cc45","Type":"ContainerDied","Data":"3b4b53991c288db565eef485b14094a28cbf38486812258ee925dd627651211a"}
Jul 24 18:10:56.051829 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.051752    2551 generic.go:356] "Generic (PLEG): container finished" podID="266cec25-978d-4bb7-8819-e2771d4d16d3" containerID="c4d9d5bdcf39c544f2415647502c93867a32fad3bb78f4a6949d7821121b40f8" exitCode=0
Jul 24 18:10:56.051829 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.051820    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2" event={"ID":"266cec25-978d-4bb7-8819-e2771d4d16d3","Type":"ContainerDied","Data":"c4d9d5bdcf39c544f2415647502c93867a32fad3bb78f4a6949d7821121b40f8"}
Jul 24 18:10:56.152572 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.152547    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:56.178532 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.178503    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:56.189994 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.189968    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:56.237882 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.237851    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pss7\" (UniqueName: \"kubernetes.io/projected/0f8147c9-0596-419b-88e3-2c3a32d3cc45-kube-api-access-4pss7\") pod \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") "
Jul 24 18:10:56.237882 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.237886    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjbsv\" (UniqueName: \"kubernetes.io/projected/54d0d18e-8c44-4f1f-a922-5252486b96b5-kube-api-access-cjbsv\") pod \"54d0d18e-8c44-4f1f-a922-5252486b96b5\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") "
Jul 24 18:10:56.238156 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.237916    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rdd7\" (UniqueName: \"kubernetes.io/projected/266cec25-978d-4bb7-8819-e2771d4d16d3-kube-api-access-9rdd7\") pod \"266cec25-978d-4bb7-8819-e2771d4d16d3\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") "
Jul 24 18:10:56.238156 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.237952    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266cec25-978d-4bb7-8819-e2771d4d16d3-openshift-service-ca\") pod \"266cec25-978d-4bb7-8819-e2771d4d16d3\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") "
Jul 24 18:10:56.238156 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.238012    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls\") pod \"266cec25-978d-4bb7-8819-e2771d4d16d3\" (UID: \"266cec25-978d-4bb7-8819-e2771d4d16d3\") "
Jul 24 18:10:56.238156 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.238039    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8147c9-0596-419b-88e3-2c3a32d3cc45-plugins-config-volume\") pod \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\" (UID: \"0f8147c9-0596-419b-88e3-2c3a32d3cc45\") "
Jul 24 18:10:56.238156 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.238117    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d0d18e-8c44-4f1f-a922-5252486b96b5-plugins-config-volume\") pod \"54d0d18e-8c44-4f1f-a922-5252486b96b5\" (UID: \"54d0d18e-8c44-4f1f-a922-5252486b96b5\") "
Jul 24 18:10:56.238537 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.238409    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266cec25-978d-4bb7-8819-e2771d4d16d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "266cec25-978d-4bb7-8819-e2771d4d16d3" (UID: "266cec25-978d-4bb7-8819-e2771d4d16d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:56.238687 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.238657    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d0d18e-8c44-4f1f-a922-5252486b96b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54d0d18e-8c44-4f1f-a922-5252486b96b5" (UID: "54d0d18e-8c44-4f1f-a922-5252486b96b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:56.238799 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.238726    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8147c9-0596-419b-88e3-2c3a32d3cc45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f8147c9-0596-419b-88e3-2c3a32d3cc45" (UID: "0f8147c9-0596-419b-88e3-2c3a32d3cc45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:10:56.240669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.240633    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d0d18e-8c44-4f1f-a922-5252486b96b5-kube-api-access-cjbsv" (OuterVolumeSpecName: "kube-api-access-cjbsv") pod "54d0d18e-8c44-4f1f-a922-5252486b96b5" (UID: "54d0d18e-8c44-4f1f-a922-5252486b96b5"). InnerVolumeSpecName "kube-api-access-cjbsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:56.240669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.240633    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8147c9-0596-419b-88e3-2c3a32d3cc45-kube-api-access-4pss7" (OuterVolumeSpecName: "kube-api-access-4pss7") pod "0f8147c9-0596-419b-88e3-2c3a32d3cc45" (UID: "0f8147c9-0596-419b-88e3-2c3a32d3cc45"). InnerVolumeSpecName "kube-api-access-4pss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:56.240856 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.240790    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266cec25-978d-4bb7-8819-e2771d4d16d3-kube-api-access-9rdd7" (OuterVolumeSpecName: "kube-api-access-9rdd7") pod "266cec25-978d-4bb7-8819-e2771d4d16d3" (UID: "266cec25-978d-4bb7-8819-e2771d4d16d3"). InnerVolumeSpecName "kube-api-access-9rdd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:10:56.240856 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.240838    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "266cec25-978d-4bb7-8819-e2771d4d16d3" (UID: "266cec25-978d-4bb7-8819-e2771d4d16d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:10:56.339923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.339835    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266cec25-978d-4bb7-8819-e2771d4d16d3-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:56.339923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.339869    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8147c9-0596-419b-88e3-2c3a32d3cc45-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:56.339923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.339879    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d0d18e-8c44-4f1f-a922-5252486b96b5-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:56.339923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.339887    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pss7\" (UniqueName: \"kubernetes.io/projected/0f8147c9-0596-419b-88e3-2c3a32d3cc45-kube-api-access-4pss7\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:56.339923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.339897    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbsv\" (UniqueName: \"kubernetes.io/projected/54d0d18e-8c44-4f1f-a922-5252486b96b5-kube-api-access-cjbsv\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:56.339923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.339905    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdd7\" (UniqueName: \"kubernetes.io/projected/266cec25-978d-4bb7-8819-e2771d4d16d3-kube-api-access-9rdd7\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:56.339923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:56.339913    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266cec25-978d-4bb7-8819-e2771d4d16d3-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:10:57.059670 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.059637    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"
Jul 24 18:10:57.060091 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.059641    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn" event={"ID":"0f8147c9-0596-419b-88e3-2c3a32d3cc45","Type":"ContainerDied","Data":"603f699e40c34bea6d16c701d09d5d796e72a9ec9ab2d8a2f32c0dc23de53908"}
Jul 24 18:10:57.060091 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.059790    2551 scope.go:117] "RemoveContainer" containerID="3b4b53991c288db565eef485b14094a28cbf38486812258ee925dd627651211a"
Jul 24 18:10:57.061195 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.061074    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"
Jul 24 18:10:57.061195 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.061117    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2" event={"ID":"266cec25-978d-4bb7-8819-e2771d4d16d3","Type":"ContainerDied","Data":"8a2b8cd92befd9b6aad1a6cf8b4a56e3c84410e1fe5f795bfddb927089d55d4a"}
Jul 24 18:10:57.062526 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.062501    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz" event={"ID":"54d0d18e-8c44-4f1f-a922-5252486b96b5","Type":"ContainerDied","Data":"a660766889dbe2210a5874443688a948939db091230bc4897db3cd680b644dff"}
Jul 24 18:10:57.062648 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.062533    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"
Jul 24 18:10:57.069178 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.069157    2551 scope.go:117] "RemoveContainer" containerID="c4d9d5bdcf39c544f2415647502c93867a32fad3bb78f4a6949d7821121b40f8"
Jul 24 18:10:57.078684 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.078661    2551 scope.go:117] "RemoveContainer" containerID="73d685569e45c045fe0b9ffbe17254051d17c426584e2e893bd2026e884df218"
Jul 24 18:10:57.089920 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.089895    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"]
Jul 24 18:10:57.093196 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.093170    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0858c257-858df494b4-t9gwz"]
Jul 24 18:10:57.104515 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.104494    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"]
Jul 24 18:10:57.109314 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.109288    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0858c257-7877f586c6-jvmcn"]
Jul 24 18:10:57.122087 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.122067    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"]
Jul 24 18:10:57.126564 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.126542    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0858c257-f67c586c5-rgtn2"]
Jul 24 18:10:57.302859 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.302820    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8147c9-0596-419b-88e3-2c3a32d3cc45" path="/var/lib/kubelet/pods/0f8147c9-0596-419b-88e3-2c3a32d3cc45/volumes"
Jul 24 18:10:57.303416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.303398    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="266cec25-978d-4bb7-8819-e2771d4d16d3" path="/var/lib/kubelet/pods/266cec25-978d-4bb7-8819-e2771d4d16d3/volumes"
Jul 24 18:10:57.303866 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:57.303851    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d0d18e-8c44-4f1f-a922-5252486b96b5" path="/var/lib/kubelet/pods/54d0d18e-8c44-4f1f-a922-5252486b96b5/volumes"
Jul 24 18:10:58.865119 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:10:58.865077    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:11:04.781747 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.781710    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"]
Jul 24 18:11:04.782161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782149    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8147c9-0596-419b-88e3-2c3a32d3cc45" containerName="payload-processing"
Jul 24 18:11:04.782210 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782164    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8147c9-0596-419b-88e3-2c3a32d3cc45" containerName="payload-processing"
Jul 24 18:11:04.782210 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782172    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266cec25-978d-4bb7-8819-e2771d4d16d3" containerName="maas-api"
Jul 24 18:11:04.782210 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782177    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="266cec25-978d-4bb7-8819-e2771d4d16d3" containerName="maas-api"
Jul 24 18:11:04.782210 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782188    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d0d18e-8c44-4f1f-a922-5252486b96b5" containerName="payload-pre-processing"
Jul 24 18:11:04.782210 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782194    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d0d18e-8c44-4f1f-a922-5252486b96b5" containerName="payload-pre-processing"
Jul 24 18:11:04.782391 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782262    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="266cec25-978d-4bb7-8819-e2771d4d16d3" containerName="maas-api"
Jul 24 18:11:04.782391 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782271    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d0d18e-8c44-4f1f-a922-5252486b96b5" containerName="payload-pre-processing"
Jul 24 18:11:04.782391 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.782276    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8147c9-0596-419b-88e3-2c3a32d3cc45" containerName="payload-processing"
Jul 24 18:11:04.787346 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.787321    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.789947 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.789917    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:11:04.790620 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.790600    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:11:04.790620 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.790611    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:11:04.790895 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.790878    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-eb6423d9-serving-cert\""
Jul 24 18:11:04.797484 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.797459    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"]
Jul 24 18:11:04.816455 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.816420    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fca9f8-013f-4bff-bb19-f692c039b8ff-openshift-service-ca\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.816773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.816744    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02fca9f8-013f-4bff-bb19-f692c039b8ff-maas-api-tls\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.816853 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.816820    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwwk\" (UniqueName: \"kubernetes.io/projected/02fca9f8-013f-4bff-bb19-f692c039b8ff-kube-api-access-lbwwk\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.918168 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.918134    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02fca9f8-013f-4bff-bb19-f692c039b8ff-maas-api-tls\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.918349 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.918181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwwk\" (UniqueName: \"kubernetes.io/projected/02fca9f8-013f-4bff-bb19-f692c039b8ff-kube-api-access-lbwwk\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.918349 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.918213    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fca9f8-013f-4bff-bb19-f692c039b8ff-openshift-service-ca\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.918931 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.918908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fca9f8-013f-4bff-bb19-f692c039b8ff-openshift-service-ca\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.920745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.920716    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02fca9f8-013f-4bff-bb19-f692c039b8ff-maas-api-tls\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:04.928565 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:04.928545    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwwk\" (UniqueName: \"kubernetes.io/projected/02fca9f8-013f-4bff-bb19-f692c039b8ff-kube-api-access-lbwwk\") pod \"maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:05.099262 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.099215    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:05.248265 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.248236    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"]
Jul 24 18:11:05.250725 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:11:05.250691    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fca9f8_013f_4bff_bb19_f692c039b8ff.slice/crio-b1e11e4db9d5fcc7c8769939d84f2b4af7df0eb0b69dee5fd75439a47e17425d WatchSource:0}: Error finding container b1e11e4db9d5fcc7c8769939d84f2b4af7df0eb0b69dee5fd75439a47e17425d: Status 404 returned error can't find the container with id b1e11e4db9d5fcc7c8769939d84f2b4af7df0eb0b69dee5fd75439a47e17425d
Jul 24 18:11:05.664326 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.664287    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"]
Jul 24 18:11:05.669305 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.669286    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:05.672209 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.672180    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-eb6423d9\""
Jul 24 18:11:05.672209 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.672201    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-eb6423d9-dockercfg-g6nq6\""
Jul 24 18:11:05.682043 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.682013    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"]
Jul 24 18:11:05.725594 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.725560    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:05.725757 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.725720    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndrv\" (UniqueName: \"kubernetes.io/projected/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-kube-api-access-6ndrv\") pod \"payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:05.790876 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.790850    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"]
Jul 24 18:11:05.794986 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.794965    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:05.807307 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.807281    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"]
Jul 24 18:11:05.826250 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.826217    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndrv\" (UniqueName: \"kubernetes.io/projected/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-kube-api-access-6ndrv\") pod \"payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:05.826471 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.826270    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:05.826936 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.826915    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:05.835437 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.835406    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndrv\" (UniqueName: \"kubernetes.io/projected/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-kube-api-access-6ndrv\") pod \"payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:05.927031 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.926943    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1691d586-0136-41be-a2b3-b224e6635f9b-plugins-config-volume\") pod \"payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") " pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:05.927186 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.927117    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsjt\" (UniqueName: \"kubernetes.io/projected/1691d586-0136-41be-a2b3-b224e6635f9b-kube-api-access-wjsjt\") pod \"payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") " pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:05.980302 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:05.980267    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:06.027990 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.027957    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsjt\" (UniqueName: \"kubernetes.io/projected/1691d586-0136-41be-a2b3-b224e6635f9b-kube-api-access-wjsjt\") pod \"payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") " pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:06.028150 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.028022    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1691d586-0136-41be-a2b3-b224e6635f9b-plugins-config-volume\") pod \"payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") " pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:06.028920 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.028884    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1691d586-0136-41be-a2b3-b224e6635f9b-plugins-config-volume\") pod \"payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") " pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:06.037770 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.037717    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsjt\" (UniqueName: \"kubernetes.io/projected/1691d586-0136-41be-a2b3-b224e6635f9b-kube-api-access-wjsjt\") pod \"payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") " pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:06.106800 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.106766    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:06.114898 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.114864    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5" event={"ID":"02fca9f8-013f-4bff-bb19-f692c039b8ff","Type":"ContainerStarted","Data":"17e00bc67e6f04b22979554bc00b23c3cd903515d2c4993a170de758283a0933"}
Jul 24 18:11:06.115057 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.114935    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:06.115057 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.114957    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5" event={"ID":"02fca9f8-013f-4bff-bb19-f692c039b8ff","Type":"ContainerStarted","Data":"b1e11e4db9d5fcc7c8769939d84f2b4af7df0eb0b69dee5fd75439a47e17425d"}
Jul 24 18:11:06.123640 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.123617    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"]
Jul 24 18:11:06.126732 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:11:06.126705    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc22a3d_33fb_4fd2_bb88_20d189d7d567.slice/crio-35ed493c31434e2ed2f725f0e255857dfb5a99bdd84500cab4375536bc51a3e8 WatchSource:0}: Error finding container 35ed493c31434e2ed2f725f0e255857dfb5a99bdd84500cab4375536bc51a3e8: Status 404 returned error can't find the container with id 35ed493c31434e2ed2f725f0e255857dfb5a99bdd84500cab4375536bc51a3e8
Jul 24 18:11:06.139526 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.139477    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5" podStartSLOduration=1.662986145 podStartE2EDuration="2.139460184s" totalImagesPullingTime="476.474039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:11:06.137658774 +0000 UTC m=+1857.254586095" watchObservedRunningTime="2026-07-24 18:11:06.139460184 +0000 UTC m=+1857.256387553"
Jul 24 18:11:06.257825 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:06.257800    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"]
Jul 24 18:11:06.259642 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:11:06.259617    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1691d586_0136_41be_a2b3_b224e6635f9b.slice/crio-cabb3dbefa0bd304b9132479e47261dba37b9a70b9dc32674d57dc3d07d65d3a WatchSource:0}: Error finding container cabb3dbefa0bd304b9132479e47261dba37b9a70b9dc32674d57dc3d07d65d3a: Status 404 returned error can't find the container with id cabb3dbefa0bd304b9132479e47261dba37b9a70b9dc32674d57dc3d07d65d3a
Jul 24 18:11:07.120861 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.120822    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9" event={"ID":"8bc22a3d-33fb-4fd2-bb88-20d189d7d567","Type":"ContainerStarted","Data":"7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb"}
Jul 24 18:11:07.121335 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.120878    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9" event={"ID":"8bc22a3d-33fb-4fd2-bb88-20d189d7d567","Type":"ContainerStarted","Data":"35ed493c31434e2ed2f725f0e255857dfb5a99bdd84500cab4375536bc51a3e8"}
Jul 24 18:11:07.121335 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.120920    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:07.122746 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.122720    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk" event={"ID":"1691d586-0136-41be-a2b3-b224e6635f9b","Type":"ContainerStarted","Data":"0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9"}
Jul 24 18:11:07.122841 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.122774    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk" event={"ID":"1691d586-0136-41be-a2b3-b224e6635f9b","Type":"ContainerStarted","Data":"cabb3dbefa0bd304b9132479e47261dba37b9a70b9dc32674d57dc3d07d65d3a"}
Jul 24 18:11:07.122901 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.122878    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:07.140527 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.140478    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9" podStartSLOduration=2.140462799 podStartE2EDuration="2.140462799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:11:07.138578092 +0000 UTC m=+1858.255505433" watchObservedRunningTime="2026-07-24 18:11:07.140462799 +0000 UTC m=+1858.257390139"
Jul 24 18:11:07.156590 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:07.156550    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk" podStartSLOduration=2.156536531 podStartE2EDuration="2.156536531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:11:07.154800091 +0000 UTC m=+1858.271727432" watchObservedRunningTime="2026-07-24 18:11:07.156536531 +0000 UTC m=+1858.273463871"
Jul 24 18:11:09.945001 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:09.944964    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:11:12.129682 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:12.129646    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:18.130424 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:18.130342    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9" podUID="8bc22a3d-33fb-4fd2-bb88-20d189d7d567" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.43:9004: connect: connection refused"
Jul 24 18:11:18.130975 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:18.130836    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:28.129037 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:28.129006    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:40.125598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.125502    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"]
Jul 24 18:11:40.126085 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.125799    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5" podUID="02fca9f8-013f-4bff-bb19-f692c039b8ff" containerName="maas-api" containerID="cri-o://17e00bc67e6f04b22979554bc00b23c3cd903515d2c4993a170de758283a0933" gracePeriod=30
Jul 24 18:11:40.193459 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.193424    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"]
Jul 24 18:11:40.193719 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.193661    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk" podUID="1691d586-0136-41be-a2b3-b224e6635f9b" containerName="payload-processing" containerID="cri-o://0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9" gracePeriod=30
Jul 24 18:11:40.199494 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.199466    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"]
Jul 24 18:11:40.199741 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.199695    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9" podUID="8bc22a3d-33fb-4fd2-bb88-20d189d7d567" containerName="payload-pre-processing" containerID="cri-o://7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb" gracePeriod=30
Jul 24 18:11:40.288959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.288929    2551 generic.go:356] "Generic (PLEG): container finished" podID="02fca9f8-013f-4bff-bb19-f692c039b8ff" containerID="17e00bc67e6f04b22979554bc00b23c3cd903515d2c4993a170de758283a0933" exitCode=0
Jul 24 18:11:40.289118 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.289005    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5" event={"ID":"02fca9f8-013f-4bff-bb19-f692c039b8ff","Type":"ContainerDied","Data":"17e00bc67e6f04b22979554bc00b23c3cd903515d2c4993a170de758283a0933"}
Jul 24 18:11:40.460908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.460882    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:40.485967 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.485938    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:40.492542 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.492519    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:40.562590 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.562557    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1691d586-0136-41be-a2b3-b224e6635f9b-plugins-config-volume\") pod \"1691d586-0136-41be-a2b3-b224e6635f9b\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") "
Jul 24 18:11:40.562773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.562617    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbwwk\" (UniqueName: \"kubernetes.io/projected/02fca9f8-013f-4bff-bb19-f692c039b8ff-kube-api-access-lbwwk\") pod \"02fca9f8-013f-4bff-bb19-f692c039b8ff\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") "
Jul 24 18:11:40.562773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.562661    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ndrv\" (UniqueName: \"kubernetes.io/projected/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-kube-api-access-6ndrv\") pod \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") "
Jul 24 18:11:40.562773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.562759    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fca9f8-013f-4bff-bb19-f692c039b8ff-openshift-service-ca\") pod \"02fca9f8-013f-4bff-bb19-f692c039b8ff\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") "
Jul 24 18:11:40.562959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.562802    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-plugins-config-volume\") pod \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\" (UID: \"8bc22a3d-33fb-4fd2-bb88-20d189d7d567\") "
Jul 24 18:11:40.562959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.562831    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02fca9f8-013f-4bff-bb19-f692c039b8ff-maas-api-tls\") pod \"02fca9f8-013f-4bff-bb19-f692c039b8ff\" (UID: \"02fca9f8-013f-4bff-bb19-f692c039b8ff\") "
Jul 24 18:11:40.562959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.562880    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjsjt\" (UniqueName: \"kubernetes.io/projected/1691d586-0136-41be-a2b3-b224e6635f9b-kube-api-access-wjsjt\") pod \"1691d586-0136-41be-a2b3-b224e6635f9b\" (UID: \"1691d586-0136-41be-a2b3-b224e6635f9b\") "
Jul 24 18:11:40.563115 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.563045    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1691d586-0136-41be-a2b3-b224e6635f9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1691d586-0136-41be-a2b3-b224e6635f9b" (UID: "1691d586-0136-41be-a2b3-b224e6635f9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:11:40.563168 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.563122    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fca9f8-013f-4bff-bb19-f692c039b8ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02fca9f8-013f-4bff-bb19-f692c039b8ff" (UID: "02fca9f8-013f-4bff-bb19-f692c039b8ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:11:40.563226 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.563161    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bc22a3d-33fb-4fd2-bb88-20d189d7d567" (UID: "8bc22a3d-33fb-4fd2-bb88-20d189d7d567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:11:40.563291 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.563275    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:11:40.563345 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.563295    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1691d586-0136-41be-a2b3-b224e6635f9b-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:11:40.563345 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.563307    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fca9f8-013f-4bff-bb19-f692c039b8ff-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:11:40.565006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.564979    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02fca9f8-013f-4bff-bb19-f692c039b8ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02fca9f8-013f-4bff-bb19-f692c039b8ff" (UID: "02fca9f8-013f-4bff-bb19-f692c039b8ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:11:40.565006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.564991    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fca9f8-013f-4bff-bb19-f692c039b8ff-kube-api-access-lbwwk" (OuterVolumeSpecName: "kube-api-access-lbwwk") pod "02fca9f8-013f-4bff-bb19-f692c039b8ff" (UID: "02fca9f8-013f-4bff-bb19-f692c039b8ff"). InnerVolumeSpecName "kube-api-access-lbwwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:11:40.565190 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.565018    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-kube-api-access-6ndrv" (OuterVolumeSpecName: "kube-api-access-6ndrv") pod "8bc22a3d-33fb-4fd2-bb88-20d189d7d567" (UID: "8bc22a3d-33fb-4fd2-bb88-20d189d7d567"). InnerVolumeSpecName "kube-api-access-6ndrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:11:40.565248 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.565231    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1691d586-0136-41be-a2b3-b224e6635f9b-kube-api-access-wjsjt" (OuterVolumeSpecName: "kube-api-access-wjsjt") pod "1691d586-0136-41be-a2b3-b224e6635f9b" (UID: "1691d586-0136-41be-a2b3-b224e6635f9b"). InnerVolumeSpecName "kube-api-access-wjsjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:11:40.664626 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.664597    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwwk\" (UniqueName: \"kubernetes.io/projected/02fca9f8-013f-4bff-bb19-f692c039b8ff-kube-api-access-lbwwk\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:11:40.664626 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.664622    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndrv\" (UniqueName: \"kubernetes.io/projected/8bc22a3d-33fb-4fd2-bb88-20d189d7d567-kube-api-access-6ndrv\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:11:40.664626 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.664632    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02fca9f8-013f-4bff-bb19-f692c039b8ff-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:11:40.664860 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:40.664641    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjsjt\" (UniqueName: \"kubernetes.io/projected/1691d586-0136-41be-a2b3-b224e6635f9b-kube-api-access-wjsjt\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:11:41.295993 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.295962    2551 generic.go:356] "Generic (PLEG): container finished" podID="8bc22a3d-33fb-4fd2-bb88-20d189d7d567" containerID="7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb" exitCode=0
Jul 24 18:11:41.296424 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.296042    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"
Jul 24 18:11:41.297599 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.297571    2551 generic.go:356] "Generic (PLEG): container finished" podID="1691d586-0136-41be-a2b3-b224e6635f9b" containerID="0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9" exitCode=0
Jul 24 18:11:41.297744 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.297626    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"
Jul 24 18:11:41.299111 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.299037    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"
Jul 24 18:11:41.302034 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.302005    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9" event={"ID":"8bc22a3d-33fb-4fd2-bb88-20d189d7d567","Type":"ContainerDied","Data":"7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb"}
Jul 24 18:11:41.302151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.302058    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9" event={"ID":"8bc22a3d-33fb-4fd2-bb88-20d189d7d567","Type":"ContainerDied","Data":"35ed493c31434e2ed2f725f0e255857dfb5a99bdd84500cab4375536bc51a3e8"}
Jul 24 18:11:41.302151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.302084    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk" event={"ID":"1691d586-0136-41be-a2b3-b224e6635f9b","Type":"ContainerDied","Data":"0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9"}
Jul 24 18:11:41.302151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.302112    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk" event={"ID":"1691d586-0136-41be-a2b3-b224e6635f9b","Type":"ContainerDied","Data":"cabb3dbefa0bd304b9132479e47261dba37b9a70b9dc32674d57dc3d07d65d3a"}
Jul 24 18:11:41.302151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.302133    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5" event={"ID":"02fca9f8-013f-4bff-bb19-f692c039b8ff","Type":"ContainerDied","Data":"b1e11e4db9d5fcc7c8769939d84f2b4af7df0eb0b69dee5fd75439a47e17425d"}
Jul 24 18:11:41.302393 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.302166    2551 scope.go:117] "RemoveContainer" containerID="7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb"
Jul 24 18:11:41.316968 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.316946    2551 scope.go:117] "RemoveContainer" containerID="7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb"
Jul 24 18:11:41.317283 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:11:41.317265    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb\": container with ID starting with 7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb not found: ID does not exist" containerID="7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb"
Jul 24 18:11:41.317358 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.317293    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb"} err="failed to get container status \"7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb\": rpc error: code = NotFound desc = could not find container \"7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb\": container with ID starting with 7d7a88aef559b3606fcbc584cff3d771efe38ef8e3089090fc2c2db169de02cb not found: ID does not exist"
Jul 24 18:11:41.317358 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.317311    2551 scope.go:117] "RemoveContainer" containerID="0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9"
Jul 24 18:11:41.325190 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.325164    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"]
Jul 24 18:11:41.327494 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.327470    2551 scope.go:117] "RemoveContainer" containerID="0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9"
Jul 24 18:11:41.327840 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:11:41.327817    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9\": container with ID starting with 0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9 not found: ID does not exist" containerID="0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9"
Jul 24 18:11:41.327944 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.327847    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9"} err="failed to get container status \"0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9\": rpc error: code = NotFound desc = could not find container \"0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9\": container with ID starting with 0082e5417a627fdf09b27b79f57d0d6e310a2e74527ad9edf06a91ddbbab19c9 not found: ID does not exist"
Jul 24 18:11:41.327944 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.327909    2551 scope.go:117] "RemoveContainer" containerID="17e00bc67e6f04b22979554bc00b23c3cd903515d2c4993a170de758283a0933"
Jul 24 18:11:41.331428 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.331402    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-eb6423d9-7b5d9b5db4-t9lc9"]
Jul 24 18:11:41.345132 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.345102    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"]
Jul 24 18:11:41.347025 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.347003    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eb6423d9-6cf9d94d9c-984q5"]
Jul 24 18:11:41.357597 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.357573    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"]
Jul 24 18:11:41.361979 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:41.361954    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-eb6423d9-7c4b447bfd-drktk"]
Jul 24 18:11:43.057567 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:43.057520    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:11:43.300516 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:43.300484    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02fca9f8-013f-4bff-bb19-f692c039b8ff" path="/var/lib/kubelet/pods/02fca9f8-013f-4bff-bb19-f692c039b8ff/volumes"
Jul 24 18:11:43.300851 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:43.300838    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1691d586-0136-41be-a2b3-b224e6635f9b" path="/var/lib/kubelet/pods/1691d586-0136-41be-a2b3-b224e6635f9b/volumes"
Jul 24 18:11:43.301142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:11:43.301131    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc22a3d-33fb-4fd2-bb88-20d189d7d567" path="/var/lib/kubelet/pods/8bc22a3d-33fb-4fd2-bb88-20d189d7d567/volumes"
Jul 24 18:13:11.441833 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:11.441797    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:13:23.668055 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:23.668018    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:13:27.092766 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.092731    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"]
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093114    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02fca9f8-013f-4bff-bb19-f692c039b8ff" containerName="maas-api"
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093125    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="02fca9f8-013f-4bff-bb19-f692c039b8ff" containerName="maas-api"
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093150    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc22a3d-33fb-4fd2-bb88-20d189d7d567" containerName="payload-pre-processing"
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093156    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc22a3d-33fb-4fd2-bb88-20d189d7d567" containerName="payload-pre-processing"
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093169    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1691d586-0136-41be-a2b3-b224e6635f9b" containerName="payload-processing"
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093174    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1691d586-0136-41be-a2b3-b224e6635f9b" containerName="payload-processing"
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093229    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc22a3d-33fb-4fd2-bb88-20d189d7d567" containerName="payload-pre-processing"
Jul 24 18:13:27.093232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093236    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="02fca9f8-013f-4bff-bb19-f692c039b8ff" containerName="maas-api"
Jul 24 18:13:27.093699 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.093245    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1691d586-0136-41be-a2b3-b224e6635f9b" containerName="payload-processing"
Jul 24 18:13:27.096550 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.096529    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.099203 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.099179    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:13:27.099203 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.099192    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:13:27.099962 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.099943    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:13:27.100036 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.099943    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8e8a9ad3-serving-cert\""
Jul 24 18:13:27.112320 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.112294    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"]
Jul 24 18:13:27.164186 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.164146    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnz76\" (UniqueName: \"kubernetes.io/projected/cfbdb788-aaee-4d54-a5e1-c49f279805b1-kube-api-access-dnz76\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.164432 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.164210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.164432 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.164261    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbdb788-aaee-4d54-a5e1-c49f279805b1-openshift-service-ca\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.265001 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.264971    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnz76\" (UniqueName: \"kubernetes.io/projected/cfbdb788-aaee-4d54-a5e1-c49f279805b1-kube-api-access-dnz76\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.265177 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.265016    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.265177 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.265055    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbdb788-aaee-4d54-a5e1-c49f279805b1-openshift-service-ca\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.265249 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:13:27.265180    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-serving-cert: secret "maas-api-e2e-mt-8e8a9ad3-serving-cert" not found
Jul 24 18:13:27.265249 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:13:27.265243    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls podName:cfbdb788-aaee-4d54-a5e1-c49f279805b1 nodeName:}" failed. No retries permitted until 2026-07-24 18:13:27.765224942 +0000 UTC m=+1998.882152264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls") pod "maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6" (UID: "cfbdb788-aaee-4d54-a5e1-c49f279805b1") : secret "maas-api-e2e-mt-8e8a9ad3-serving-cert" not found
Jul 24 18:13:27.265683 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.265665    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbdb788-aaee-4d54-a5e1-c49f279805b1-openshift-service-ca\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.286812 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.286787    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnz76\" (UniqueName: \"kubernetes.io/projected/cfbdb788-aaee-4d54-a5e1-c49f279805b1-kube-api-access-dnz76\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.769232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.769190    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:27.771793 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:27.771761    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls\") pod \"maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:28.008579 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.008542    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:28.042048 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.042014    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"]
Jul 24 18:13:28.045874 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.045852    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.048697 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.048677    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8e8a9ad3-dockercfg-8rc7h\""
Jul 24 18:13:28.048916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.048901    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8e8a9ad3\""
Jul 24 18:13:28.060036 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.060007    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"]
Jul 24 18:13:28.071454 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.071418    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.071593 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.071488    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5gl\" (UniqueName: \"kubernetes.io/projected/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-kube-api-access-2v5gl\") pod \"payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.139897 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.139861    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"]
Jul 24 18:13:28.143559 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.143536    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.160996 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.160968    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"]
Jul 24 18:13:28.168824 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.168795    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"]
Jul 24 18:13:28.172278 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.172116    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90fac42-abee-4643-bde6-523843333318-plugins-config-volume\") pod \"payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") " pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.172278 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.172205    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.172278 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.172249    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsb7r\" (UniqueName: \"kubernetes.io/projected/f90fac42-abee-4643-bde6-523843333318-kube-api-access-zsb7r\") pod \"payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") " pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.172278 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.172278    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5gl\" (UniqueName: \"kubernetes.io/projected/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-kube-api-access-2v5gl\") pod \"payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.172732 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:13:28.172709    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbdb788_aaee_4d54_a5e1_c49f279805b1.slice/crio-af722ceb899054d2506f46af4235fab8af242eb83da181b739c00310e5bef7a4 WatchSource:0}: Error finding container af722ceb899054d2506f46af4235fab8af242eb83da181b739c00310e5bef7a4: Status 404 returned error can't find the container with id af722ceb899054d2506f46af4235fab8af242eb83da181b739c00310e5bef7a4
Jul 24 18:13:28.173013 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.172993    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.184576 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.184548    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5gl\" (UniqueName: \"kubernetes.io/projected/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-kube-api-access-2v5gl\") pod \"payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.273604 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.273507    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90fac42-abee-4643-bde6-523843333318-plugins-config-volume\") pod \"payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") " pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.273764 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.273613    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsb7r\" (UniqueName: \"kubernetes.io/projected/f90fac42-abee-4643-bde6-523843333318-kube-api-access-zsb7r\") pod \"payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") " pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.274244 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.274223    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90fac42-abee-4643-bde6-523843333318-plugins-config-volume\") pod \"payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") " pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.283735 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.283706    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsb7r\" (UniqueName: \"kubernetes.io/projected/f90fac42-abee-4643-bde6-523843333318-kube-api-access-zsb7r\") pod \"payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") " pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.358446 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.358409    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.454998 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.454727    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.524941 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.524915    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"]
Jul 24 18:13:28.526895 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:13:28.526867    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c791e4_40b2_4ebd_b143_c093f6a72a1b.slice/crio-8f41dd9a39ca293a7ba55c3d2a05d4c9fbf723366d11deecc6dce749156d2dea WatchSource:0}: Error finding container 8f41dd9a39ca293a7ba55c3d2a05d4c9fbf723366d11deecc6dce749156d2dea: Status 404 returned error can't find the container with id 8f41dd9a39ca293a7ba55c3d2a05d4c9fbf723366d11deecc6dce749156d2dea
Jul 24 18:13:28.625496 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.625469    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"]
Jul 24 18:13:28.633600 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:13:28.633569    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90fac42_abee_4643_bde6_523843333318.slice/crio-4528dea7e9a2d0843c415b03f01afb7169d5fa4138aaf14f5033cbcf3df699d5 WatchSource:0}: Error finding container 4528dea7e9a2d0843c415b03f01afb7169d5fa4138aaf14f5033cbcf3df699d5: Status 404 returned error can't find the container with id 4528dea7e9a2d0843c415b03f01afb7169d5fa4138aaf14f5033cbcf3df699d5
Jul 24 18:13:28.823205 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.823166    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6" event={"ID":"cfbdb788-aaee-4d54-a5e1-c49f279805b1","Type":"ContainerStarted","Data":"252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db"}
Jul 24 18:13:28.823416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.823270    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:28.823416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.823310    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6" event={"ID":"cfbdb788-aaee-4d54-a5e1-c49f279805b1","Type":"ContainerStarted","Data":"af722ceb899054d2506f46af4235fab8af242eb83da181b739c00310e5bef7a4"}
Jul 24 18:13:28.824764 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.824743    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx" event={"ID":"a9c791e4-40b2-4ebd-b143-c093f6a72a1b","Type":"ContainerStarted","Data":"eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932"}
Jul 24 18:13:28.824854 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.824777    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx" event={"ID":"a9c791e4-40b2-4ebd-b143-c093f6a72a1b","Type":"ContainerStarted","Data":"8f41dd9a39ca293a7ba55c3d2a05d4c9fbf723366d11deecc6dce749156d2dea"}
Jul 24 18:13:28.824854 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.824844    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:28.826315 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.826293    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs" event={"ID":"f90fac42-abee-4643-bde6-523843333318","Type":"ContainerStarted","Data":"19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f"}
Jul 24 18:13:28.826411 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.826329    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs" event={"ID":"f90fac42-abee-4643-bde6-523843333318","Type":"ContainerStarted","Data":"4528dea7e9a2d0843c415b03f01afb7169d5fa4138aaf14f5033cbcf3df699d5"}
Jul 24 18:13:28.826411 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.826357    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:28.866093 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.866043    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6" podStartSLOduration=1.437791657 podStartE2EDuration="1.866029001s" totalImagesPullingTime="428.237344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:28.86410415 +0000 UTC m=+1999.981031491" watchObservedRunningTime="2026-07-24 18:13:28.866029001 +0000 UTC m=+1999.982956341"
Jul 24 18:13:28.895052 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.894900    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx" podStartSLOduration=0.894879402 podStartE2EDuration="894.879402ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:28.893051807 +0000 UTC m=+2000.009979149" watchObservedRunningTime="2026-07-24 18:13:28.894879402 +0000 UTC m=+2000.011806746"
Jul 24 18:13:28.927982 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:28.927929    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs" podStartSLOduration=0.927911947 podStartE2EDuration="927.911947ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:28.927298253 +0000 UTC m=+2000.044225593" watchObservedRunningTime="2026-07-24 18:13:28.927911947 +0000 UTC m=+2000.044839328"
Jul 24 18:13:34.835749 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:34.835716    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:36.852080 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:36.852045    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:13:39.832164 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:39.832131    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:39.832614 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:39.832216    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:49.256891 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.256842    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"]
Jul 24 18:13:49.257415 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.257080    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6" podUID="cfbdb788-aaee-4d54-a5e1-c49f279805b1" containerName="maas-api" containerID="cri-o://252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db" gracePeriod=30
Jul 24 18:13:49.331526 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.331488    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"]
Jul 24 18:13:49.331765 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.331737    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs" podUID="f90fac42-abee-4643-bde6-523843333318" containerName="payload-processing" containerID="cri-o://19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f" gracePeriod=30
Jul 24 18:13:49.339533 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.339313    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"]
Jul 24 18:13:49.339684 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.339543    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx" podUID="a9c791e4-40b2-4ebd-b143-c093f6a72a1b" containerName="payload-pre-processing" containerID="cri-o://eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932" gracePeriod=30
Jul 24 18:13:49.610801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.610773    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:49.639862 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.639837    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:49.643006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.642983    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:49.661177 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.661152    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsb7r\" (UniqueName: \"kubernetes.io/projected/f90fac42-abee-4643-bde6-523843333318-kube-api-access-zsb7r\") pod \"f90fac42-abee-4643-bde6-523843333318\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") "
Jul 24 18:13:49.661329 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.661216    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90fac42-abee-4643-bde6-523843333318-plugins-config-volume\") pod \"f90fac42-abee-4643-bde6-523843333318\" (UID: \"f90fac42-abee-4643-bde6-523843333318\") "
Jul 24 18:13:49.661694 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.661671    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90fac42-abee-4643-bde6-523843333318-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f90fac42-abee-4643-bde6-523843333318" (UID: "f90fac42-abee-4643-bde6-523843333318"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:13:49.663444 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.663415    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90fac42-abee-4643-bde6-523843333318-kube-api-access-zsb7r" (OuterVolumeSpecName: "kube-api-access-zsb7r") pod "f90fac42-abee-4643-bde6-523843333318" (UID: "f90fac42-abee-4643-bde6-523843333318"). InnerVolumeSpecName "kube-api-access-zsb7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:13:49.762036 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.761994    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls\") pod \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") "
Jul 24 18:13:49.762230 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762087    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbdb788-aaee-4d54-a5e1-c49f279805b1-openshift-service-ca\") pod \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") "
Jul 24 18:13:49.762230 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762138    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-plugins-config-volume\") pod \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") "
Jul 24 18:13:49.762230 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762165    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v5gl\" (UniqueName: \"kubernetes.io/projected/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-kube-api-access-2v5gl\") pod \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\" (UID: \"a9c791e4-40b2-4ebd-b143-c093f6a72a1b\") "
Jul 24 18:13:49.762230 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762202    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnz76\" (UniqueName: \"kubernetes.io/projected/cfbdb788-aaee-4d54-a5e1-c49f279805b1-kube-api-access-dnz76\") pod \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\" (UID: \"cfbdb788-aaee-4d54-a5e1-c49f279805b1\") "
Jul 24 18:13:49.762520 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762490    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9c791e4-40b2-4ebd-b143-c093f6a72a1b" (UID: "a9c791e4-40b2-4ebd-b143-c093f6a72a1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:13:49.762520 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762504    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbdb788-aaee-4d54-a5e1-c49f279805b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfbdb788-aaee-4d54-a5e1-c49f279805b1" (UID: "cfbdb788-aaee-4d54-a5e1-c49f279805b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:13:49.762628 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762574    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsb7r\" (UniqueName: \"kubernetes.io/projected/f90fac42-abee-4643-bde6-523843333318-kube-api-access-zsb7r\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:13:49.762628 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.762593    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90fac42-abee-4643-bde6-523843333318-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:13:49.764298 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.764278    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfbdb788-aaee-4d54-a5e1-c49f279805b1" (UID: "cfbdb788-aaee-4d54-a5e1-c49f279805b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:13:49.764424 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.764316    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbdb788-aaee-4d54-a5e1-c49f279805b1-kube-api-access-dnz76" (OuterVolumeSpecName: "kube-api-access-dnz76") pod "cfbdb788-aaee-4d54-a5e1-c49f279805b1" (UID: "cfbdb788-aaee-4d54-a5e1-c49f279805b1"). InnerVolumeSpecName "kube-api-access-dnz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:13:49.764424 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.764320    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-kube-api-access-2v5gl" (OuterVolumeSpecName: "kube-api-access-2v5gl") pod "a9c791e4-40b2-4ebd-b143-c093f6a72a1b" (UID: "a9c791e4-40b2-4ebd-b143-c093f6a72a1b"). InnerVolumeSpecName "kube-api-access-2v5gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:13:49.863191 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.863160    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:13:49.863191 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.863188    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v5gl\" (UniqueName: \"kubernetes.io/projected/a9c791e4-40b2-4ebd-b143-c093f6a72a1b-kube-api-access-2v5gl\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:13:49.863191 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.863197    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnz76\" (UniqueName: \"kubernetes.io/projected/cfbdb788-aaee-4d54-a5e1-c49f279805b1-kube-api-access-dnz76\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:13:49.863470 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.863207    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfbdb788-aaee-4d54-a5e1-c49f279805b1-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:13:49.863470 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.863215    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfbdb788-aaee-4d54-a5e1-c49f279805b1-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:13:49.926440 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.926402    2551 generic.go:356] "Generic (PLEG): container finished" podID="cfbdb788-aaee-4d54-a5e1-c49f279805b1" containerID="252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db" exitCode=0
Jul 24 18:13:49.926592 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.926441    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"
Jul 24 18:13:49.926592 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.926469    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6" event={"ID":"cfbdb788-aaee-4d54-a5e1-c49f279805b1","Type":"ContainerDied","Data":"252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db"}
Jul 24 18:13:49.926592 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.926528    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6" event={"ID":"cfbdb788-aaee-4d54-a5e1-c49f279805b1","Type":"ContainerDied","Data":"af722ceb899054d2506f46af4235fab8af242eb83da181b739c00310e5bef7a4"}
Jul 24 18:13:49.926592 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.926561    2551 scope.go:117] "RemoveContainer" containerID="252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db"
Jul 24 18:13:49.927966 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.927941    2551 generic.go:356] "Generic (PLEG): container finished" podID="a9c791e4-40b2-4ebd-b143-c093f6a72a1b" containerID="eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932" exitCode=0
Jul 24 18:13:49.928115 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.927981    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"
Jul 24 18:13:49.928115 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.927972    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx" event={"ID":"a9c791e4-40b2-4ebd-b143-c093f6a72a1b","Type":"ContainerDied","Data":"eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932"}
Jul 24 18:13:49.928115 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.928070    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx" event={"ID":"a9c791e4-40b2-4ebd-b143-c093f6a72a1b","Type":"ContainerDied","Data":"8f41dd9a39ca293a7ba55c3d2a05d4c9fbf723366d11deecc6dce749156d2dea"}
Jul 24 18:13:49.929509 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.929486    2551 generic.go:356] "Generic (PLEG): container finished" podID="f90fac42-abee-4643-bde6-523843333318" containerID="19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f" exitCode=0
Jul 24 18:13:49.929630 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.929516    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs" event={"ID":"f90fac42-abee-4643-bde6-523843333318","Type":"ContainerDied","Data":"19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f"}
Jul 24 18:13:49.929630 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.929532    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"
Jul 24 18:13:49.929630 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.929566    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs" event={"ID":"f90fac42-abee-4643-bde6-523843333318","Type":"ContainerDied","Data":"4528dea7e9a2d0843c415b03f01afb7169d5fa4138aaf14f5033cbcf3df699d5"}
Jul 24 18:13:49.937465 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.937447    2551 scope.go:117] "RemoveContainer" containerID="252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db"
Jul 24 18:13:49.938215 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:13:49.938190    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db\": container with ID starting with 252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db not found: ID does not exist" containerID="252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db"
Jul 24 18:13:49.938318 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.938224    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db"} err="failed to get container status \"252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db\": rpc error: code = NotFound desc = could not find container \"252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db\": container with ID starting with 252c15d33f1bdde4f824f4aa039d8519ed6fe19aa1d5f8832c658292fb8d42db not found: ID does not exist"
Jul 24 18:13:49.938318 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.938252    2551 scope.go:117] "RemoveContainer" containerID="eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932"
Jul 24 18:13:49.953089 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.953071    2551 scope.go:117] "RemoveContainer" containerID="eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932"
Jul 24 18:13:49.953411 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:13:49.953364    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932\": container with ID starting with eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932 not found: ID does not exist" containerID="eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932"
Jul 24 18:13:49.953521 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.953421    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932"} err="failed to get container status \"eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932\": rpc error: code = NotFound desc = could not find container \"eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932\": container with ID starting with eff0162363f271751a42d2a6634b7b7c1adfb2bc8e8cd5e929153372a0df5932 not found: ID does not exist"
Jul 24 18:13:49.953521 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.953445    2551 scope.go:117] "RemoveContainer" containerID="19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f"
Jul 24 18:13:49.956824 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.956805    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"]
Jul 24 18:13:49.962054 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.962032    2551 scope.go:117] "RemoveContainer" containerID="19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f"
Jul 24 18:13:49.962329 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:13:49.962310    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f\": container with ID starting with 19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f not found: ID does not exist" containerID="19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f"
Jul 24 18:13:49.962395 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.962338    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f"} err="failed to get container status \"19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f\": rpc error: code = NotFound desc = could not find container \"19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f\": container with ID starting with 19fcace2f52c2fde7a6ab99b219e96513aa571a1071efafde238e6079a057a6f not found: ID does not exist"
Jul 24 18:13:49.963431 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.963363    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8e8a9ad3-5f848c97d5-vfkj6"]
Jul 24 18:13:49.976349 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.976328    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"]
Jul 24 18:13:49.982446 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.982426    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8e8a9ad3-6d7fd6998d-dv9zx"]
Jul 24 18:13:49.996457 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:49.996431    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"]
Jul 24 18:13:50.000328 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:50.000301    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8e8a9ad3-679f7b7766-f2prs"]
Jul 24 18:13:51.300333 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:51.300295    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9c791e4-40b2-4ebd-b143-c093f6a72a1b" path="/var/lib/kubelet/pods/a9c791e4-40b2-4ebd-b143-c093f6a72a1b/volumes"
Jul 24 18:13:51.300722 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:51.300700    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfbdb788-aaee-4d54-a5e1-c49f279805b1" path="/var/lib/kubelet/pods/cfbdb788-aaee-4d54-a5e1-c49f279805b1/volumes"
Jul 24 18:13:51.300998 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:51.300985    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90fac42-abee-4643-bde6-523843333318" path="/var/lib/kubelet/pods/f90fac42-abee-4643-bde6-523843333318/volumes"
Jul 24 18:13:52.359729 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:13:52.359693    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:14:23.140670 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.140632    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"]
Jul 24 18:14:23.141483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141459    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90fac42-abee-4643-bde6-523843333318" containerName="payload-processing"
Jul 24 18:14:23.141483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141484    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90fac42-abee-4643-bde6-523843333318" containerName="payload-processing"
Jul 24 18:14:23.141669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141515    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbdb788-aaee-4d54-a5e1-c49f279805b1" containerName="maas-api"
Jul 24 18:14:23.141669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141523    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbdb788-aaee-4d54-a5e1-c49f279805b1" containerName="maas-api"
Jul 24 18:14:23.141669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141538    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c791e4-40b2-4ebd-b143-c093f6a72a1b" containerName="payload-pre-processing"
Jul 24 18:14:23.141669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141547    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c791e4-40b2-4ebd-b143-c093f6a72a1b" containerName="payload-pre-processing"
Jul 24 18:14:23.141669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141642    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c791e4-40b2-4ebd-b143-c093f6a72a1b" containerName="payload-pre-processing"
Jul 24 18:14:23.141669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141660    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90fac42-abee-4643-bde6-523843333318" containerName="payload-processing"
Jul 24 18:14:23.141977 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.141673    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbdb788-aaee-4d54-a5e1-c49f279805b1" containerName="maas-api"
Jul 24 18:14:23.145178 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.145154    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.148862 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.148837    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ad8b31a2-serving-cert\""
Jul 24 18:14:23.148962 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.148849    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:14:23.149542 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.149526    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:14:23.149633 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.149542    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:14:23.157320 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.157296    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"]
Jul 24 18:14:23.165786 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.165759    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-openshift-service-ca\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.165912 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.165876    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthcg\" (UniqueName: \"kubernetes.io/projected/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-kube-api-access-lthcg\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.165977 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.165930    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.266755 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.266720    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthcg\" (UniqueName: \"kubernetes.io/projected/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-kube-api-access-lthcg\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.266935 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.266769    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.266935 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.266840    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-openshift-service-ca\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.267010 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:14:23.266960    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-serving-cert: secret "maas-api-e2e-mt-ad8b31a2-serving-cert" not found
Jul 24 18:14:23.267054 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:14:23.267042    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls podName:90db73a3-02e6-4d80-8ed3-df2bc09a67c9 nodeName:}" failed. No retries permitted until 2026-07-24 18:14:23.767023586 +0000 UTC m=+2054.883950907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls") pod "maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb" (UID: "90db73a3-02e6-4d80-8ed3-df2bc09a67c9") : secret "maas-api-e2e-mt-ad8b31a2-serving-cert" not found
Jul 24 18:14:23.267538 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.267521    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-openshift-service-ca\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.279101 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.279072    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthcg\" (UniqueName: \"kubernetes.io/projected/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-kube-api-access-lthcg\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.771666 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.771629    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.774181 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.774152    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls\") pod \"maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:23.968315 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.968284    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"]
Jul 24 18:14:23.972168 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.972151    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:23.975813 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.975786    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ad8b31a2\""
Jul 24 18:14:23.975994 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.975980    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ad8b31a2-dockercfg-4rhlt\""
Jul 24 18:14:23.989691 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:23.989657    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"]
Jul 24 18:14:24.016034 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.015999    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"]
Jul 24 18:14:24.019648 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.019624    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.030729 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.030702    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"]
Jul 24 18:14:24.058344 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.058315    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:24.074829 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.074798    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de1b48c-3582-4198-9ead-963ebc4bc3d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:24.074971 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.074870    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llj6\" (UniqueName: \"kubernetes.io/projected/0de1b48c-3582-4198-9ead-963ebc4bc3d2-kube-api-access-5llj6\") pod \"payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:24.074971 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.074938    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f802ff-7230-474c-96c5-e58247a314be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.075091 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.075013    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczrm\" (UniqueName: \"kubernetes.io/projected/e8f802ff-7230-474c-96c5-e58247a314be-kube-api-access-hczrm\") pod \"payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.175913 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.175828    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llj6\" (UniqueName: \"kubernetes.io/projected/0de1b48c-3582-4198-9ead-963ebc4bc3d2-kube-api-access-5llj6\") pod \"payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:24.175913 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.175889    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f802ff-7230-474c-96c5-e58247a314be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.176827 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.175935    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczrm\" (UniqueName: \"kubernetes.io/projected/e8f802ff-7230-474c-96c5-e58247a314be-kube-api-access-hczrm\") pod \"payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.176827 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.175969    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de1b48c-3582-4198-9ead-963ebc4bc3d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:24.176827 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.176618    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de1b48c-3582-4198-9ead-963ebc4bc3d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:24.177070 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.177047    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f802ff-7230-474c-96c5-e58247a314be-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.185025 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.185003    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczrm\" (UniqueName: \"kubernetes.io/projected/e8f802ff-7230-474c-96c5-e58247a314be-kube-api-access-hczrm\") pod \"payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.185419 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.185370    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llj6\" (UniqueName: \"kubernetes.io/projected/0de1b48c-3582-4198-9ead-963ebc4bc3d2-kube-api-access-5llj6\") pod \"payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") " pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:24.207334 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.207307    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"]
Jul 24 18:14:24.209170 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:14:24.209142    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90db73a3_02e6_4d80_8ed3_df2bc09a67c9.slice/crio-d1e21689ac37247b132fb7713c57d4363ca96114cfcd5ca1f35f412a6cb20387 WatchSource:0}: Error finding container d1e21689ac37247b132fb7713c57d4363ca96114cfcd5ca1f35f412a6cb20387: Status 404 returned error can't find the container with id d1e21689ac37247b132fb7713c57d4363ca96114cfcd5ca1f35f412a6cb20387
Jul 24 18:14:24.210706 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.210688    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:14:24.283148 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.283072    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:24.330169 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.330127    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:24.441108 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.441081    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"]
Jul 24 18:14:24.444022 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:14:24.443990    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de1b48c_3582_4198_9ead_963ebc4bc3d2.slice/crio-96b2e9430b4d8bcdd44e0f259dd1a970b90c768c6304c8ee4f0de6d2eee1195e WatchSource:0}: Error finding container 96b2e9430b4d8bcdd44e0f259dd1a970b90c768c6304c8ee4f0de6d2eee1195e: Status 404 returned error can't find the container with id 96b2e9430b4d8bcdd44e0f259dd1a970b90c768c6304c8ee4f0de6d2eee1195e
Jul 24 18:14:24.484345 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:24.484314    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"]
Jul 24 18:14:24.492694 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:14:24.492664    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f802ff_7230_474c_96c5_e58247a314be.slice/crio-2bb43898b3c52622fe67717a33cabd4036f3a520947485fff9c8cb70147ca527 WatchSource:0}: Error finding container 2bb43898b3c52622fe67717a33cabd4036f3a520947485fff9c8cb70147ca527: Status 404 returned error can't find the container with id 2bb43898b3c52622fe67717a33cabd4036f3a520947485fff9c8cb70147ca527
Jul 24 18:14:25.111416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.111291    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb" event={"ID":"0de1b48c-3582-4198-9ead-963ebc4bc3d2","Type":"ContainerStarted","Data":"8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9"}
Jul 24 18:14:25.111416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.111354    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb" event={"ID":"0de1b48c-3582-4198-9ead-963ebc4bc3d2","Type":"ContainerStarted","Data":"96b2e9430b4d8bcdd44e0f259dd1a970b90c768c6304c8ee4f0de6d2eee1195e"}
Jul 24 18:14:25.111416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.111405    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:25.112969 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.112938    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn" event={"ID":"e8f802ff-7230-474c-96c5-e58247a314be","Type":"ContainerStarted","Data":"6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6"}
Jul 24 18:14:25.113113 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.112994    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn" event={"ID":"e8f802ff-7230-474c-96c5-e58247a314be","Type":"ContainerStarted","Data":"2bb43898b3c52622fe67717a33cabd4036f3a520947485fff9c8cb70147ca527"}
Jul 24 18:14:25.113113 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.113048    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:25.114414 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.114364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb" event={"ID":"90db73a3-02e6-4d80-8ed3-df2bc09a67c9","Type":"ContainerStarted","Data":"9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157"}
Jul 24 18:14:25.114514 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.114428    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb" event={"ID":"90db73a3-02e6-4d80-8ed3-df2bc09a67c9","Type":"ContainerStarted","Data":"d1e21689ac37247b132fb7713c57d4363ca96114cfcd5ca1f35f412a6cb20387"}
Jul 24 18:14:25.114567 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.114529    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:25.148140 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.148092    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb" podStartSLOduration=2.148075923 podStartE2EDuration="2.148075923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:14:25.144425215 +0000 UTC m=+2056.261352584" watchObservedRunningTime="2026-07-24 18:14:25.148075923 +0000 UTC m=+2056.265003262"
Jul 24 18:14:25.176812 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.176738    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb" podStartSLOduration=1.567455148 podStartE2EDuration="2.176721233s" totalImagesPullingTime="609.266085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:14:25.173451047 +0000 UTC m=+2056.290378391" watchObservedRunningTime="2026-07-24 18:14:25.176721233 +0000 UTC m=+2056.293648572"
Jul 24 18:14:25.202697 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:25.202642    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn" podStartSLOduration=2.202625849 podStartE2EDuration="2.202625849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:14:25.200676958 +0000 UTC m=+2056.317604298" watchObservedRunningTime="2026-07-24 18:14:25.202625849 +0000 UTC m=+2056.319553192"
Jul 24 18:14:28.486594 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:28.486551    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:14:31.133068 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:31.132992    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:14:36.128815 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.128780    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:14:36.129282 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.128861    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:14:36.613055 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.613023    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"]
Jul 24 18:14:36.618239 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.618214    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.621795 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.621761    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ad8b31a2\"/\"openshift-service-ca.crt\""
Jul 24 18:14:36.621795 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.621790    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ad8b31a2\"/\"default-dockercfg-w9pjx\""
Jul 24 18:14:36.622004 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.621761    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ad8b31a2\"/\"e2e-lifecycle-model-ad8b31a2-kserve-self-signed-certs\""
Jul 24 18:14:36.622004 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.621798    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ad8b31a2\"/\"kube-root-ca.crt\""
Jul 24 18:14:36.628269 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.628247    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"]
Jul 24 18:14:36.695181 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.695143    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjrq\" (UniqueName: \"kubernetes.io/projected/c280cc57-3e63-4d04-b035-0b7ce006d25c-kube-api-access-ffjrq\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.695426 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.695205    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-model-cache\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.695426 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.695274    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-home\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.695426 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.695334    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-dshm\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.695426 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.695399    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c280cc57-3e63-4d04-b035-0b7ce006d25c-tls-certs\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.695655 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.695492    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-tmp-dir\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.796568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.796539    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-tmp-dir\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.796568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.796572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjrq\" (UniqueName: \"kubernetes.io/projected/c280cc57-3e63-4d04-b035-0b7ce006d25c-kube-api-access-ffjrq\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.796813 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.796616    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-model-cache\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.796813 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.796634    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-home\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.796813 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.796662    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-dshm\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.796813 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.796688    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c280cc57-3e63-4d04-b035-0b7ce006d25c-tls-certs\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.797059 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.797031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-tmp-dir\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.797126 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.797037    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-model-cache\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.797178 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.797123    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-home\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.799051 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.799031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-dshm\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.799386 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.799353    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c280cc57-3e63-4d04-b035-0b7ce006d25c-tls-certs\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.805168 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.805149    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjrq\" (UniqueName: \"kubernetes.io/projected/c280cc57-3e63-4d04-b035-0b7ce006d25c-kube-api-access-ffjrq\") pod \"e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") " pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:36.929568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:36.929466    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:37.068523 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:37.068491    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"]
Jul 24 18:14:37.071044 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:14:37.071016    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc280cc57_3e63_4d04_b035_0b7ce006d25c.slice/crio-2d8b9c9ba017f910e6dcf5c21552f46568404039e9188013c631f310f62747c0 WatchSource:0}: Error finding container 2d8b9c9ba017f910e6dcf5c21552f46568404039e9188013c631f310f62747c0: Status 404 returned error can't find the container with id 2d8b9c9ba017f910e6dcf5c21552f46568404039e9188013c631f310f62747c0
Jul 24 18:14:37.183226 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:37.183148    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw" event={"ID":"c280cc57-3e63-4d04-b035-0b7ce006d25c","Type":"ContainerStarted","Data":"800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a"}
Jul 24 18:14:37.183226 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:37.183188    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw" event={"ID":"c280cc57-3e63-4d04-b035-0b7ce006d25c","Type":"ContainerStarted","Data":"2d8b9c9ba017f910e6dcf5c21552f46568404039e9188013c631f310f62747c0"}
Jul 24 18:14:37.202743 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:37.202685    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw" podStartSLOduration=1.202670313 podStartE2EDuration="1.202670313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:14:37.201499679 +0000 UTC m=+2068.318427021" watchObservedRunningTime="2026-07-24 18:14:37.202670313 +0000 UTC m=+2068.319597675"
Jul 24 18:14:39.945798 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:39.945756    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:14:46.930432 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:46.930386    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:46.930937 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:46.930454    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:46.943025 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:46.942999    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:14:47.245239 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:14:47.245156    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:15:09.378014 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:09.377988    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:15:09.384590 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:09.384571    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:15:18.194714 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:18.194664    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:15:37.253603 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:37.253570    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:15:38.496132 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.496089    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"]
Jul 24 18:15:38.497582 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.496968    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb" podUID="90db73a3-02e6-4d80-8ed3-df2bc09a67c9" containerName="maas-api" containerID="cri-o://9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157" gracePeriod=30
Jul 24 18:15:38.560364 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.560322    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"]
Jul 24 18:15:38.560691 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.560624    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb" podUID="0de1b48c-3582-4198-9ead-963ebc4bc3d2" containerName="payload-processing" containerID="cri-o://8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9" gracePeriod=30
Jul 24 18:15:38.567763 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.567728    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"]
Jul 24 18:15:38.568074 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.567958    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn" podUID="e8f802ff-7230-474c-96c5-e58247a314be" containerName="payload-pre-processing" containerID="cri-o://6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6" gracePeriod=30
Jul 24 18:15:38.876934 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.876909    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:15:38.879665 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.879643    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:15:38.883003 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.882983    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:15:38.888324 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.888305    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-openshift-service-ca\") pod \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") "
Jul 24 18:15:38.888460 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.888329    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls\") pod \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") "
Jul 24 18:15:38.888460 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.888394    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lthcg\" (UniqueName: \"kubernetes.io/projected/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-kube-api-access-lthcg\") pod \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\" (UID: \"90db73a3-02e6-4d80-8ed3-df2bc09a67c9\") "
Jul 24 18:15:38.888460 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.888419    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de1b48c-3582-4198-9ead-963ebc4bc3d2-plugins-config-volume\") pod \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") "
Jul 24 18:15:38.888639 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.888463    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5llj6\" (UniqueName: \"kubernetes.io/projected/0de1b48c-3582-4198-9ead-963ebc4bc3d2-kube-api-access-5llj6\") pod \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\" (UID: \"0de1b48c-3582-4198-9ead-963ebc4bc3d2\") "
Jul 24 18:15:38.888712 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.888683    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90db73a3-02e6-4d80-8ed3-df2bc09a67c9" (UID: "90db73a3-02e6-4d80-8ed3-df2bc09a67c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:15:38.888830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.888800    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de1b48c-3582-4198-9ead-963ebc4bc3d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0de1b48c-3582-4198-9ead-963ebc4bc3d2" (UID: "0de1b48c-3582-4198-9ead-963ebc4bc3d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:15:38.890645 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.890617    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90db73a3-02e6-4d80-8ed3-df2bc09a67c9" (UID: "90db73a3-02e6-4d80-8ed3-df2bc09a67c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:15:38.890727 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.890649    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de1b48c-3582-4198-9ead-963ebc4bc3d2-kube-api-access-5llj6" (OuterVolumeSpecName: "kube-api-access-5llj6") pod "0de1b48c-3582-4198-9ead-963ebc4bc3d2" (UID: "0de1b48c-3582-4198-9ead-963ebc4bc3d2"). InnerVolumeSpecName "kube-api-access-5llj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:15:38.890727 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.890697    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-kube-api-access-lthcg" (OuterVolumeSpecName: "kube-api-access-lthcg") pod "90db73a3-02e6-4d80-8ed3-df2bc09a67c9" (UID: "90db73a3-02e6-4d80-8ed3-df2bc09a67c9"). InnerVolumeSpecName "kube-api-access-lthcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:15:38.989165 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989132    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hczrm\" (UniqueName: \"kubernetes.io/projected/e8f802ff-7230-474c-96c5-e58247a314be-kube-api-access-hczrm\") pod \"e8f802ff-7230-474c-96c5-e58247a314be\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") "
Jul 24 18:15:38.989339 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989204    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f802ff-7230-474c-96c5-e58247a314be-plugins-config-volume\") pod \"e8f802ff-7230-474c-96c5-e58247a314be\" (UID: \"e8f802ff-7230-474c-96c5-e58247a314be\") "
Jul 24 18:15:38.989409 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989396    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:15:38.989451 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989412    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:15:38.989451 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989426    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lthcg\" (UniqueName: \"kubernetes.io/projected/90db73a3-02e6-4d80-8ed3-df2bc09a67c9-kube-api-access-lthcg\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:15:38.989451 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989440    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de1b48c-3582-4198-9ead-963ebc4bc3d2-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:15:38.989545 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989452    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llj6\" (UniqueName: \"kubernetes.io/projected/0de1b48c-3582-4198-9ead-963ebc4bc3d2-kube-api-access-5llj6\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:15:38.989595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.989562    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f802ff-7230-474c-96c5-e58247a314be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8f802ff-7230-474c-96c5-e58247a314be" (UID: "e8f802ff-7230-474c-96c5-e58247a314be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:15:38.991497 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:38.991470    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f802ff-7230-474c-96c5-e58247a314be-kube-api-access-hczrm" (OuterVolumeSpecName: "kube-api-access-hczrm") pod "e8f802ff-7230-474c-96c5-e58247a314be" (UID: "e8f802ff-7230-474c-96c5-e58247a314be"). InnerVolumeSpecName "kube-api-access-hczrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:15:39.090784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.090744    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hczrm\" (UniqueName: \"kubernetes.io/projected/e8f802ff-7230-474c-96c5-e58247a314be-kube-api-access-hczrm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:15:39.090784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.090784    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f802ff-7230-474c-96c5-e58247a314be-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:15:39.493292 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.493207    2551 generic.go:356] "Generic (PLEG): container finished" podID="e8f802ff-7230-474c-96c5-e58247a314be" containerID="6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6" exitCode=0
Jul 24 18:15:39.493292 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.493252    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"
Jul 24 18:15:39.493569 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.493292    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn" event={"ID":"e8f802ff-7230-474c-96c5-e58247a314be","Type":"ContainerDied","Data":"6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6"}
Jul 24 18:15:39.493569 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.493343    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn" event={"ID":"e8f802ff-7230-474c-96c5-e58247a314be","Type":"ContainerDied","Data":"2bb43898b3c52622fe67717a33cabd4036f3a520947485fff9c8cb70147ca527"}
Jul 24 18:15:39.493569 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.493403    2551 scope.go:117] "RemoveContainer" containerID="6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6"
Jul 24 18:15:39.494879 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.494852    2551 generic.go:356] "Generic (PLEG): container finished" podID="90db73a3-02e6-4d80-8ed3-df2bc09a67c9" containerID="9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157" exitCode=0
Jul 24 18:15:39.495050 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.494917    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"
Jul 24 18:15:39.495127 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.494991    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb" event={"ID":"90db73a3-02e6-4d80-8ed3-df2bc09a67c9","Type":"ContainerDied","Data":"9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157"}
Jul 24 18:15:39.495188 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.495173    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb" event={"ID":"90db73a3-02e6-4d80-8ed3-df2bc09a67c9","Type":"ContainerDied","Data":"d1e21689ac37247b132fb7713c57d4363ca96114cfcd5ca1f35f412a6cb20387"}
Jul 24 18:15:39.496542 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.496517    2551 generic.go:356] "Generic (PLEG): container finished" podID="0de1b48c-3582-4198-9ead-963ebc4bc3d2" containerID="8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9" exitCode=0
Jul 24 18:15:39.496881 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.496555    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"
Jul 24 18:15:39.496881 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.496571    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb" event={"ID":"0de1b48c-3582-4198-9ead-963ebc4bc3d2","Type":"ContainerDied","Data":"8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9"}
Jul 24 18:15:39.496881 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.496610    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb" event={"ID":"0de1b48c-3582-4198-9ead-963ebc4bc3d2","Type":"ContainerDied","Data":"96b2e9430b4d8bcdd44e0f259dd1a970b90c768c6304c8ee4f0de6d2eee1195e"}
Jul 24 18:15:39.504196 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.504044    2551 scope.go:117] "RemoveContainer" containerID="6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6"
Jul 24 18:15:39.504344 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:15:39.504323    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6\": container with ID starting with 6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6 not found: ID does not exist" containerID="6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6"
Jul 24 18:15:39.504413 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.504354    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6"} err="failed to get container status \"6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6\": rpc error: code = NotFound desc = could not find container \"6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6\": container with ID starting with 6cf74cc2d961c97eb77963404981dea106dcaec54ef473628a7d3141f0f302f6 not found: ID does not exist"
Jul 24 18:15:39.504413 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.504386    2551 scope.go:117] "RemoveContainer" containerID="9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157"
Jul 24 18:15:39.517326 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.517303    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"]
Jul 24 18:15:39.521172 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.521149    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ad8b31a2-5c8fbb6846-cgpnb"]
Jul 24 18:15:39.522541 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.522520    2551 scope.go:117] "RemoveContainer" containerID="9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157"
Jul 24 18:15:39.522851 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:15:39.522831    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157\": container with ID starting with 9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157 not found: ID does not exist" containerID="9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157"
Jul 24 18:15:39.522937 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.522860    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157"} err="failed to get container status \"9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157\": rpc error: code = NotFound desc = could not find container \"9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157\": container with ID starting with 9db9ef2eaef1a447b716f2c0bcfe6c4462da1c3fe93d00ae99ff33e81509b157 not found: ID does not exist"
Jul 24 18:15:39.522937 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.522884    2551 scope.go:117] "RemoveContainer" containerID="8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9"
Jul 24 18:15:39.531618 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.531501    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"]
Jul 24 18:15:39.531689 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.531659    2551 scope.go:117] "RemoveContainer" containerID="8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9"
Jul 24 18:15:39.531969 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:15:39.531948    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9\": container with ID starting with 8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9 not found: ID does not exist" containerID="8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9"
Jul 24 18:15:39.532036 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.531988    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9"} err="failed to get container status \"8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9\": rpc error: code = NotFound desc = could not find container \"8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9\": container with ID starting with 8519d84ab872b23fcc6625bfbdc8b25e5a6678acd176b785cea230358c4fbfc9 not found: ID does not exist"
Jul 24 18:15:39.537548 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.537526    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad8b31a2-847db58c86-jzlsb"]
Jul 24 18:15:39.550888 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.550868    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"]
Jul 24 18:15:39.555926 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:39.555899    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad8b31a2-59c7fcd4f6-qspmn"]
Jul 24 18:15:41.300495 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:41.300461    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de1b48c-3582-4198-9ead-963ebc4bc3d2" path="/var/lib/kubelet/pods/0de1b48c-3582-4198-9ead-963ebc4bc3d2/volumes"
Jul 24 18:15:41.300881 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:41.300858    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90db73a3-02e6-4d80-8ed3-df2bc09a67c9" path="/var/lib/kubelet/pods/90db73a3-02e6-4d80-8ed3-df2bc09a67c9/volumes"
Jul 24 18:15:41.301159 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:41.301147    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f802ff-7230-474c-96c5-e58247a314be" path="/var/lib/kubelet/pods/e8f802ff-7230-474c-96c5-e58247a314be/volumes"
Jul 24 18:15:48.460397 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:48.460350    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:15:51.247131 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:15:51.247082    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"]
Jul 24 18:16:06.273917 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.273860    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw" podUID="c280cc57-3e63-4d04-b035-0b7ce006d25c" containerName="main" containerID="cri-o://800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a" gracePeriod=45
Jul 24 18:16:06.511930 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.511906    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:16:06.631749 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.631716    2551 generic.go:356] "Generic (PLEG): container finished" podID="c280cc57-3e63-4d04-b035-0b7ce006d25c" containerID="800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a" exitCode=0
Jul 24 18:16:06.631925 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.631779    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"
Jul 24 18:16:06.631925 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.631787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw" event={"ID":"c280cc57-3e63-4d04-b035-0b7ce006d25c","Type":"ContainerDied","Data":"800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a"}
Jul 24 18:16:06.631925 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.631826    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw" event={"ID":"c280cc57-3e63-4d04-b035-0b7ce006d25c","Type":"ContainerDied","Data":"2d8b9c9ba017f910e6dcf5c21552f46568404039e9188013c631f310f62747c0"}
Jul 24 18:16:06.631925 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.631852    2551 scope.go:117] "RemoveContainer" containerID="800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a"
Jul 24 18:16:06.640369 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.640347    2551 scope.go:117] "RemoveContainer" containerID="800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a"
Jul 24 18:16:06.640702 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:16:06.640667    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a\": container with ID starting with 800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a not found: ID does not exist" containerID="800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a"
Jul 24 18:16:06.640792 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.640711    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a"} err="failed to get container status \"800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a\": rpc error: code = NotFound desc = could not find container \"800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a\": container with ID starting with 800fd0ab31b07af5b46b3c519f727985c396dcde591a3f87727fb7e5f2e4264a not found: ID does not exist"
Jul 24 18:16:06.643962 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.643946    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c280cc57-3e63-4d04-b035-0b7ce006d25c-tls-certs\") pod \"c280cc57-3e63-4d04-b035-0b7ce006d25c\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") "
Jul 24 18:16:06.644030 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.643994    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-dshm\") pod \"c280cc57-3e63-4d04-b035-0b7ce006d25c\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") "
Jul 24 18:16:06.644072 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644032    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffjrq\" (UniqueName: \"kubernetes.io/projected/c280cc57-3e63-4d04-b035-0b7ce006d25c-kube-api-access-ffjrq\") pod \"c280cc57-3e63-4d04-b035-0b7ce006d25c\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") "
Jul 24 18:16:06.644072 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644048    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-tmp-dir\") pod \"c280cc57-3e63-4d04-b035-0b7ce006d25c\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") "
Jul 24 18:16:06.644163 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644087    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-home\") pod \"c280cc57-3e63-4d04-b035-0b7ce006d25c\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") "
Jul 24 18:16:06.644430 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644232    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-model-cache\") pod \"c280cc57-3e63-4d04-b035-0b7ce006d25c\" (UID: \"c280cc57-3e63-4d04-b035-0b7ce006d25c\") "
Jul 24 18:16:06.644430 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644332    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c280cc57-3e63-4d04-b035-0b7ce006d25c" (UID: "c280cc57-3e63-4d04-b035-0b7ce006d25c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:16:06.644430 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644344    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-home" (OuterVolumeSpecName: "home") pod "c280cc57-3e63-4d04-b035-0b7ce006d25c" (UID: "c280cc57-3e63-4d04-b035-0b7ce006d25c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:16:06.644630 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644444    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c280cc57-3e63-4d04-b035-0b7ce006d25c" (UID: "c280cc57-3e63-4d04-b035-0b7ce006d25c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:16:06.644692 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644628    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:16:06.644692 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644646    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:16:06.644692 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.644662    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:16:06.646266 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.646239    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-dshm" (OuterVolumeSpecName: "dshm") pod "c280cc57-3e63-4d04-b035-0b7ce006d25c" (UID: "c280cc57-3e63-4d04-b035-0b7ce006d25c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:16:06.646405 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.646354    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c280cc57-3e63-4d04-b035-0b7ce006d25c-kube-api-access-ffjrq" (OuterVolumeSpecName: "kube-api-access-ffjrq") pod "c280cc57-3e63-4d04-b035-0b7ce006d25c" (UID: "c280cc57-3e63-4d04-b035-0b7ce006d25c"). InnerVolumeSpecName "kube-api-access-ffjrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:16:06.646405 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.646392    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c280cc57-3e63-4d04-b035-0b7ce006d25c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c280cc57-3e63-4d04-b035-0b7ce006d25c" (UID: "c280cc57-3e63-4d04-b035-0b7ce006d25c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:16:06.745364 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.745329    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffjrq\" (UniqueName: \"kubernetes.io/projected/c280cc57-3e63-4d04-b035-0b7ce006d25c-kube-api-access-ffjrq\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:16:06.745364 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.745358    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c280cc57-3e63-4d04-b035-0b7ce006d25c-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:16:06.745364 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.745369    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c280cc57-3e63-4d04-b035-0b7ce006d25c-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:16:06.954962 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.954921    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"]
Jul 24 18:16:06.956719 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:06.956689    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ad8b31a2/e2e-lifecycle-model-ad8b31a2-kserve-55dd464b7f-fc6pw"]
Jul 24 18:16:07.302277 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:07.302246    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c280cc57-3e63-4d04-b035-0b7ce006d25c" path="/var/lib/kubelet/pods/c280cc57-3e63-4d04-b035-0b7ce006d25c/volumes"
Jul 24 18:16:26.274417 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:26.274359    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:16:35.981512 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:16:35.981471    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:17:22.042499 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042465    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"]
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042838    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f802ff-7230-474c-96c5-e58247a314be" containerName="payload-pre-processing"
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042849    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f802ff-7230-474c-96c5-e58247a314be" containerName="payload-pre-processing"
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042861    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c280cc57-3e63-4d04-b035-0b7ce006d25c" containerName="main"
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042866    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c280cc57-3e63-4d04-b035-0b7ce006d25c" containerName="main"
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042875    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de1b48c-3582-4198-9ead-963ebc4bc3d2" containerName="payload-processing"
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042881    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de1b48c-3582-4198-9ead-963ebc4bc3d2" containerName="payload-processing"
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042900    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90db73a3-02e6-4d80-8ed3-df2bc09a67c9" containerName="maas-api"
Jul 24 18:17:22.042908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042905    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="90db73a3-02e6-4d80-8ed3-df2bc09a67c9" containerName="maas-api"
Jul 24 18:17:22.043161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042952    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f802ff-7230-474c-96c5-e58247a314be" containerName="payload-pre-processing"
Jul 24 18:17:22.043161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042959    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="90db73a3-02e6-4d80-8ed3-df2bc09a67c9" containerName="maas-api"
Jul 24 18:17:22.043161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042966    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de1b48c-3582-4198-9ead-963ebc4bc3d2" containerName="payload-processing"
Jul 24 18:17:22.043161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.042974    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c280cc57-3e63-4d04-b035-0b7ce006d25c" containerName="main"
Jul 24 18:17:22.045774 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.045749    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.050089 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.050058    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6e3964-serving-cert\""
Jul 24 18:17:22.050209 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.050151    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:17:22.050412 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.050370    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:17:22.051224 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.051205    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:17:22.058559 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.058538    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"]
Jul 24 18:17:22.152691 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.152644    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0a346c-1682-4809-9c28-35885ff70ccc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.152879 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.152788    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrhq8\" (UniqueName: \"kubernetes.io/projected/bf0a346c-1682-4809-9c28-35885ff70ccc-kube-api-access-mrhq8\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.152879 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.152834    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.253595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.253553    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrhq8\" (UniqueName: \"kubernetes.io/projected/bf0a346c-1682-4809-9c28-35885ff70ccc-kube-api-access-mrhq8\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.253595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.253598    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.253818 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:17:22.253709    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-serving-cert: secret "maas-api-e2e-inf-a-6e3964-serving-cert" not found
Jul 24 18:17:22.253818 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:17:22.253769    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls podName:bf0a346c-1682-4809-9c28-35885ff70ccc nodeName:}" failed. No retries permitted until 2026-07-24 18:17:22.753751982 +0000 UTC m=+2233.870679301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls") pod "maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544" (UID: "bf0a346c-1682-4809-9c28-35885ff70ccc") : secret "maas-api-e2e-inf-a-6e3964-serving-cert" not found
Jul 24 18:17:22.253818 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.253783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0a346c-1682-4809-9c28-35885ff70ccc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.254342 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.254322    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0a346c-1682-4809-9c28-35885ff70ccc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.264715 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.264681    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrhq8\" (UniqueName: \"kubernetes.io/projected/bf0a346c-1682-4809-9c28-35885ff70ccc-kube-api-access-mrhq8\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.758544 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.758509    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.761132 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.761107    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.786115 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.786074    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"]
Jul 24 18:17:22.790082 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.790055    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:22.792535 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.792510    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6e3964\""
Jul 24 18:17:22.798320 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.798289    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"]
Jul 24 18:17:22.816734 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.816700    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"]
Jul 24 18:17:22.819941 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.819924    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:22.823551 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.823528    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6e3964-dockercfg-7r826\""
Jul 24 18:17:22.833854 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.833799    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"]
Jul 24 18:17:22.859368 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.859337    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npzz\" (UniqueName: \"kubernetes.io/projected/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-kube-api-access-7npzz\") pod \"payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:22.859607 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.859512    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:22.957954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.957912    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:22.960310 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.960280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5j6b\" (UniqueName: \"kubernetes.io/projected/14242431-a3f2-4bba-9433-850e73872f3c-kube-api-access-z5j6b\") pod \"payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:22.960469 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.960340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npzz\" (UniqueName: \"kubernetes.io/projected/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-kube-api-access-7npzz\") pod \"payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:22.960469 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.960365    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14242431-a3f2-4bba-9433-850e73872f3c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:22.960579 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.960470    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:22.961109 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.961088    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:22.970024 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:22.970000    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npzz\" (UniqueName: \"kubernetes.io/projected/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-kube-api-access-7npzz\") pod \"payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:23.061232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.061198    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5j6b\" (UniqueName: \"kubernetes.io/projected/14242431-a3f2-4bba-9433-850e73872f3c-kube-api-access-z5j6b\") pod \"payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:23.061669 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.061266    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14242431-a3f2-4bba-9433-850e73872f3c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:23.061956 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.061937    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14242431-a3f2-4bba-9433-850e73872f3c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:23.070196 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.070165    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5j6b\" (UniqueName: \"kubernetes.io/projected/14242431-a3f2-4bba-9433-850e73872f3c-kube-api-access-z5j6b\") pod \"payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:23.101644 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.101616    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"]
Jul 24 18:17:23.102893 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.102873    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:23.104024 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:17:23.103995    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0a346c_1682_4809_9c28_35885ff70ccc.slice/crio-f3773db50394eb9cba9c204923be6f04b7b1cd551fe2d1c874823ef8b8ec451d WatchSource:0}: Error finding container f3773db50394eb9cba9c204923be6f04b7b1cd551fe2d1c874823ef8b8ec451d: Status 404 returned error can't find the container with id f3773db50394eb9cba9c204923be6f04b7b1cd551fe2d1c874823ef8b8ec451d
Jul 24 18:17:23.131149 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.131122    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:23.253322 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.253298    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"]
Jul 24 18:17:23.256743 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:17:23.256711    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bc03a7_a6a3_48b3_a600_09d0ca1e5a88.slice/crio-c5237ba71ee1078fb8421a38c4e5ec7c0c432b258d9b83643f72cfcc1003318a WatchSource:0}: Error finding container c5237ba71ee1078fb8421a38c4e5ec7c0c432b258d9b83643f72cfcc1003318a: Status 404 returned error can't find the container with id c5237ba71ee1078fb8421a38c4e5ec7c0c432b258d9b83643f72cfcc1003318a
Jul 24 18:17:23.290492 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:23.290275    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"]
Jul 24 18:17:23.293280 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:17:23.293245    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14242431_a3f2_4bba_9433_850e73872f3c.slice/crio-0bdb9fd0f926a20a83ef47e371abdba4d4a362c6e4d3fb76214b5fcd8b727f5d WatchSource:0}: Error finding container 0bdb9fd0f926a20a83ef47e371abdba4d4a362c6e4d3fb76214b5fcd8b727f5d: Status 404 returned error can't find the container with id 0bdb9fd0f926a20a83ef47e371abdba4d4a362c6e4d3fb76214b5fcd8b727f5d
Jul 24 18:17:24.002704 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.002663    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z" event={"ID":"14242431-a3f2-4bba-9433-850e73872f3c","Type":"ContainerStarted","Data":"ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7"}
Jul 24 18:17:24.002916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.002734    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:24.002916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.002758    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z" event={"ID":"14242431-a3f2-4bba-9433-850e73872f3c","Type":"ContainerStarted","Data":"0bdb9fd0f926a20a83ef47e371abdba4d4a362c6e4d3fb76214b5fcd8b727f5d"}
Jul 24 18:17:24.004463 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.004432    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc" event={"ID":"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88","Type":"ContainerStarted","Data":"ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032"}
Jul 24 18:17:24.004595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.004486    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc" event={"ID":"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88","Type":"ContainerStarted","Data":"c5237ba71ee1078fb8421a38c4e5ec7c0c432b258d9b83643f72cfcc1003318a"}
Jul 24 18:17:24.004595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.004516    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:24.006055 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.006033    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544" event={"ID":"bf0a346c-1682-4809-9c28-35885ff70ccc","Type":"ContainerStarted","Data":"72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9"}
Jul 24 18:17:24.006191 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.006070    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544" event={"ID":"bf0a346c-1682-4809-9c28-35885ff70ccc","Type":"ContainerStarted","Data":"f3773db50394eb9cba9c204923be6f04b7b1cd551fe2d1c874823ef8b8ec451d"}
Jul 24 18:17:24.006191 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.006167    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:24.020689 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.020643    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z" podStartSLOduration=2.020626963 podStartE2EDuration="2.020626963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:17:24.02032611 +0000 UTC m=+2235.137253452" watchObservedRunningTime="2026-07-24 18:17:24.020626963 +0000 UTC m=+2235.137554301"
Jul 24 18:17:24.037941 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.037884    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc" podStartSLOduration=2.037864873 podStartE2EDuration="2.037864873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:17:24.03720436 +0000 UTC m=+2235.154131702" watchObservedRunningTime="2026-07-24 18:17:24.037864873 +0000 UTC m=+2235.154792213"
Jul 24 18:17:24.059238 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:24.059182    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544" podStartSLOduration=1.6551892609999999 podStartE2EDuration="2.059166594s" totalImagesPullingTime="403.977333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:17:24.057300834 +0000 UTC m=+2235.174228166" watchObservedRunningTime="2026-07-24 18:17:24.059166594 +0000 UTC m=+2235.176093933"
Jul 24 18:17:28.386455 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:28.386411    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:17:30.017216 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.017142    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:17:30.300025 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.299943    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"]
Jul 24 18:17:30.305270 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.305245    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.307702 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.307677    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0f1d5b-serving-cert\""
Jul 24 18:17:30.321423 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.321368    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"]
Jul 24 18:17:30.431865 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.431827    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a46397-2514-46ac-b94a-f0fa18d9d717-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.432030 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.431878    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbfn\" (UniqueName: \"kubernetes.io/projected/c1a46397-2514-46ac-b94a-f0fa18d9d717-kube-api-access-9xbfn\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.432030 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.431944    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.532702 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.532668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a46397-2514-46ac-b94a-f0fa18d9d717-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.532948 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.532720    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbfn\" (UniqueName: \"kubernetes.io/projected/c1a46397-2514-46ac-b94a-f0fa18d9d717-kube-api-access-9xbfn\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.532948 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.532740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.532948 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:17:30.532874    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-serving-cert: secret "maas-api-e2e-inf-b-0f1d5b-serving-cert" not found
Jul 24 18:17:30.532948 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:17:30.532928    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls podName:c1a46397-2514-46ac-b94a-f0fa18d9d717 nodeName:}" failed. No retries permitted until 2026-07-24 18:17:31.032912257 +0000 UTC m=+2242.149839576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls") pod "maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs" (UID: "c1a46397-2514-46ac-b94a-f0fa18d9d717") : secret "maas-api-e2e-inf-b-0f1d5b-serving-cert" not found
Jul 24 18:17:30.533316 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.533294    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a46397-2514-46ac-b94a-f0fa18d9d717-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:30.549331 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:30.549296    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbfn\" (UniqueName: \"kubernetes.io/projected/c1a46397-2514-46ac-b94a-f0fa18d9d717-kube-api-access-9xbfn\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:31.037158 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.037125    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:31.039805 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.039774    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls\") pod \"maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:31.215544 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.215511    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:31.339661 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.339627    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"]
Jul 24 18:17:31.344405 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.344364    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.349614 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.349545    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0f1d5b\""
Jul 24 18:17:31.349853 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.349652    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0f1d5b-dockercfg-bxp46\""
Jul 24 18:17:31.370589 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.370557    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"]
Jul 24 18:17:31.387214 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.386861    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"]
Jul 24 18:17:31.391459 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:17:31.391426    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a46397_2514_46ac_b94a_f0fa18d9d717.slice/crio-c2283356f01d0edc7a5774a07643a6b96f8b380803584734fe6a0c4fc1e6be45 WatchSource:0}: Error finding container c2283356f01d0edc7a5774a07643a6b96f8b380803584734fe6a0c4fc1e6be45: Status 404 returned error can't find the container with id c2283356f01d0edc7a5774a07643a6b96f8b380803584734fe6a0c4fc1e6be45
Jul 24 18:17:31.410073 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.410044    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"]
Jul 24 18:17:31.413759 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.413739    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.433746 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.433663    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"]
Jul 24 18:17:31.441255 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.441229    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7d2ea2-9c43-438d-a360-35b99d527349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.441415 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.441330    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntnm\" (UniqueName: \"kubernetes.io/projected/ac7d2ea2-9c43-438d-a360-35b99d527349-kube-api-access-mntnm\") pod \"payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.542821 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.542782    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnds8\" (UniqueName: \"kubernetes.io/projected/16664846-4bc9-4e22-befe-11ccd41f0d7d-kube-api-access-dnds8\") pod \"payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.542821 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.542824    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16664846-4bc9-4e22-befe-11ccd41f0d7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.543068 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.542853    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7d2ea2-9c43-438d-a360-35b99d527349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.543068 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.542997    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntnm\" (UniqueName: \"kubernetes.io/projected/ac7d2ea2-9c43-438d-a360-35b99d527349-kube-api-access-mntnm\") pod \"payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.543616 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.543595    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7d2ea2-9c43-438d-a360-35b99d527349-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.553069 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.553042    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntnm\" (UniqueName: \"kubernetes.io/projected/ac7d2ea2-9c43-438d-a360-35b99d527349-kube-api-access-mntnm\") pod \"payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.643904 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.643807    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnds8\" (UniqueName: \"kubernetes.io/projected/16664846-4bc9-4e22-befe-11ccd41f0d7d-kube-api-access-dnds8\") pod \"payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.644086 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.643918    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16664846-4bc9-4e22-befe-11ccd41f0d7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.644620 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.644596    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16664846-4bc9-4e22-befe-11ccd41f0d7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.656126 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.656102    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:31.662828 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.662805    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnds8\" (UniqueName: \"kubernetes.io/projected/16664846-4bc9-4e22-befe-11ccd41f0d7d-kube-api-access-dnds8\") pod \"payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.724661 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.724473    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:31.836157 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.836122    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"]
Jul 24 18:17:31.837716 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:17:31.837668    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7d2ea2_9c43_438d_a360_35b99d527349.slice/crio-5f8a7c9a446b986773a56e8dc8e11cdf5ce9ce2c8d1c84f5c332a5684f1f0dbb WatchSource:0}: Error finding container 5f8a7c9a446b986773a56e8dc8e11cdf5ce9ce2c8d1c84f5c332a5684f1f0dbb: Status 404 returned error can't find the container with id 5f8a7c9a446b986773a56e8dc8e11cdf5ce9ce2c8d1c84f5c332a5684f1f0dbb
Jul 24 18:17:31.887505 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:31.887477    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"]
Jul 24 18:17:31.890972 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:17:31.890947    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16664846_4bc9_4e22_befe_11ccd41f0d7d.slice/crio-883b86f8b3aa6dfb430a40517fe572c8a37e0bfea391d4a236464cbcab14d172 WatchSource:0}: Error finding container 883b86f8b3aa6dfb430a40517fe572c8a37e0bfea391d4a236464cbcab14d172: Status 404 returned error can't find the container with id 883b86f8b3aa6dfb430a40517fe572c8a37e0bfea391d4a236464cbcab14d172
Jul 24 18:17:32.052453 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.052407    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs" event={"ID":"c1a46397-2514-46ac-b94a-f0fa18d9d717","Type":"ContainerStarted","Data":"f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc"}
Jul 24 18:17:32.052916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.052474    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:32.052916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.052501    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs" event={"ID":"c1a46397-2514-46ac-b94a-f0fa18d9d717","Type":"ContainerStarted","Data":"c2283356f01d0edc7a5774a07643a6b96f8b380803584734fe6a0c4fc1e6be45"}
Jul 24 18:17:32.054128 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.054101    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76" event={"ID":"16664846-4bc9-4e22-befe-11ccd41f0d7d","Type":"ContainerStarted","Data":"bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325"}
Jul 24 18:17:32.054262 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.054146    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76" event={"ID":"16664846-4bc9-4e22-befe-11ccd41f0d7d","Type":"ContainerStarted","Data":"883b86f8b3aa6dfb430a40517fe572c8a37e0bfea391d4a236464cbcab14d172"}
Jul 24 18:17:32.054262 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.054194    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:32.055675 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.055647    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns" event={"ID":"ac7d2ea2-9c43-438d-a360-35b99d527349","Type":"ContainerStarted","Data":"93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b"}
Jul 24 18:17:32.055772 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.055697    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:32.055772 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.055718    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns" event={"ID":"ac7d2ea2-9c43-438d-a360-35b99d527349","Type":"ContainerStarted","Data":"5f8a7c9a446b986773a56e8dc8e11cdf5ce9ce2c8d1c84f5c332a5684f1f0dbb"}
Jul 24 18:17:32.089288 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.089154    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs" podStartSLOduration=1.782114302 podStartE2EDuration="2.089133173s" totalImagesPullingTime="307.018871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:17:32.087825873 +0000 UTC m=+2243.204753224" watchObservedRunningTime="2026-07-24 18:17:32.089133173 +0000 UTC m=+2243.206060514"
Jul 24 18:17:32.114995 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.114939    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76" podStartSLOduration=1.114922552 podStartE2EDuration="1.114922552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:17:32.113760672 +0000 UTC m=+2243.230688011" watchObservedRunningTime="2026-07-24 18:17:32.114922552 +0000 UTC m=+2243.231849892"
Jul 24 18:17:32.152815 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:32.152718    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns" podStartSLOduration=1.152701425 podStartE2EDuration="1.152701425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:17:32.15061544 +0000 UTC m=+2243.267542791" watchObservedRunningTime="2026-07-24 18:17:32.152701425 +0000 UTC m=+2243.269628765"
Jul 24 18:17:35.013232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:35.013199    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:17:35.013664 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:35.013364    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:17:38.065864 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:38.065835    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:17:41.548315 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:41.548258    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:17:43.062697 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:43.062664    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:17:43.063249 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:43.062751    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:17:46.702085 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.702037    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"]
Jul 24 18:17:46.705758 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.705733    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.708169 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.708143    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6e3964\"/\"openshift-service-ca.crt\""
Jul 24 18:17:46.709127 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.709099    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6e3964\"/\"test-model-6e3964-kserve-self-signed-certs\""
Jul 24 18:17:46.709242 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.709101    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6e3964\"/\"default-dockercfg-p46dv\""
Jul 24 18:17:46.709242 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.709148    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6e3964\"/\"kube-root-ca.crt\""
Jul 24 18:17:46.714545 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.714521    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"]
Jul 24 18:17:46.812246 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.812197    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-model-cache\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.812472 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.812341    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljqp\" (UniqueName: \"kubernetes.io/projected/5f058d4f-555d-476c-b4b9-84d4894c307c-kube-api-access-rljqp\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.812472 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.812417    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-tmp-dir\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.812472 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.812445    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-dshm\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.812472 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.812469    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f058d4f-555d-476c-b4b9-84d4894c307c-tls-certs\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.812694 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.812579    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-home\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913259 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913218    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-dshm\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913456 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913269    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f058d4f-555d-476c-b4b9-84d4894c307c-tls-certs\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913456 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913330    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-home\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913456 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913396    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-model-cache\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913489    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljqp\" (UniqueName: \"kubernetes.io/projected/5f058d4f-555d-476c-b4b9-84d4894c307c-kube-api-access-rljqp\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-tmp-dir\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913797 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913768    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-model-cache\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913857 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913820    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-home\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.913857 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.913842    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-tmp-dir\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.915802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.915772    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-dshm\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.916028 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.916011    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f058d4f-555d-476c-b4b9-84d4894c307c-tls-certs\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:46.923137 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:46.923111    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljqp\" (UniqueName: \"kubernetes.io/projected/5f058d4f-555d-476c-b4b9-84d4894c307c-kube-api-access-rljqp\") pod \"test-model-6e3964-kserve-86575cc7c9-k726j\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") " pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:47.018185 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:47.018085    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:47.163913 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:47.163889    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"]
Jul 24 18:17:47.166341 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:17:47.166304    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f058d4f_555d_476c_b4b9_84d4894c307c.slice/crio-f7ff62fafa8e4d6bc60b3ae203bae94f52806f1eb8a780582b29c6673149e634 WatchSource:0}: Error finding container f7ff62fafa8e4d6bc60b3ae203bae94f52806f1eb8a780582b29c6673149e634: Status 404 returned error can't find the container with id f7ff62fafa8e4d6bc60b3ae203bae94f52806f1eb8a780582b29c6673149e634
Jul 24 18:17:48.138313 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:48.138275    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j" event={"ID":"5f058d4f-555d-476c-b4b9-84d4894c307c","Type":"ContainerStarted","Data":"74e9ca8968bfeaab4a4b30ab2892473802520eb4811b76505aec322d4b58d634"}
Jul 24 18:17:48.138724 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:48.138323    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j" event={"ID":"5f058d4f-555d-476c-b4b9-84d4894c307c","Type":"ContainerStarted","Data":"f7ff62fafa8e4d6bc60b3ae203bae94f52806f1eb8a780582b29c6673149e634"}
Jul 24 18:17:48.159116 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:48.159067    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j" podStartSLOduration=2.159050828 podStartE2EDuration="2.159050828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:17:48.15805935 +0000 UTC m=+2259.274986690" watchObservedRunningTime="2026-07-24 18:17:48.159050828 +0000 UTC m=+2259.275978168"
Jul 24 18:17:50.856785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:50.856736    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:17:57.018509 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:57.018471    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:57.018898 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:57.018629    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:57.031326 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:57.031300    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:17:57.198163 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:17:57.198135    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:18:02.742248 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:02.742211    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:18:19.511587 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.511548    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"]
Jul 24 18:18:19.515901 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.515874    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.519098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.519072    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0f1d5b\"/\"openshift-service-ca.crt\""
Jul 24 18:18:19.519098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.519095    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0f1d5b\"/\"kube-root-ca.crt\""
Jul 24 18:18:19.519307 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.519076    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0f1d5b\"/\"default-dockercfg-jvhxr\""
Jul 24 18:18:19.519307 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.519076    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0f1d5b\"/\"test-model-0f1d5b-kserve-self-signed-certs\""
Jul 24 18:18:19.525479 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.525456    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"]
Jul 24 18:18:19.529786 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.529761    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-model-cache\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.529893 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.529821    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-dshm\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.529893 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.529874    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tls-certs\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.529967 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.529903    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tmp-dir\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.529967 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.529935    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwjq\" (UniqueName: \"kubernetes.io/projected/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-kube-api-access-xtwjq\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.530034 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.530002    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-home\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.630908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.630876    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-home\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631107 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.630925    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-model-cache\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631107 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.630970    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-dshm\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631107 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.630989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tls-certs\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631107 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.631006    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tmp-dir\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631107 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.631033    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwjq\" (UniqueName: \"kubernetes.io/projected/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-kube-api-access-xtwjq\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.631320    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-home\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631490 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.631445    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-model-cache\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.631490 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.631461    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tmp-dir\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.633355 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.633333    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-dshm\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.633613 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.633593    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tls-certs\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.638868 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.638848    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwjq\" (UniqueName: \"kubernetes.io/projected/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-kube-api-access-xtwjq\") pod \"test-model-0f1d5b-kserve-7475677dc7-tvhgk\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") " pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.827193 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.827161    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:19.969142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:19.969068    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"]
Jul 24 18:18:19.971480 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:18:19.971442    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1039a87_e1eb_4e89_81a9_0f9acb4c4141.slice/crio-90d2dc7e02c928328ce3545d7e9f60997d7f8eddabbb2cefb7660da282d7024b WatchSource:0}: Error finding container 90d2dc7e02c928328ce3545d7e9f60997d7f8eddabbb2cefb7660da282d7024b: Status 404 returned error can't find the container with id 90d2dc7e02c928328ce3545d7e9f60997d7f8eddabbb2cefb7660da282d7024b
Jul 24 18:18:20.304649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:20.304607    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk" event={"ID":"d1039a87-e1eb-4e89-81a9-0f9acb4c4141","Type":"ContainerStarted","Data":"96a8d342098c88d8a113cb7cf58aea91ec7a4f80d51206ff258d0f1e7545d183"}
Jul 24 18:18:20.304832 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:20.304671    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk" event={"ID":"d1039a87-e1eb-4e89-81a9-0f9acb4c4141","Type":"ContainerStarted","Data":"90d2dc7e02c928328ce3545d7e9f60997d7f8eddabbb2cefb7660da282d7024b"}
Jul 24 18:18:20.324628 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:20.324501    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk" podStartSLOduration=1.324479749 podStartE2EDuration="1.324479749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:20.324227705 +0000 UTC m=+2291.441155046" watchObservedRunningTime="2026-07-24 18:18:20.324479749 +0000 UTC m=+2291.441407090"
Jul 24 18:18:23.654255 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:23.654209    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:18:29.828065 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:29.828025    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:29.828493 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:29.828313    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:29.841192 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:29.841167    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:30.364014 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:30.363987    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:18:37.754157 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:37.754120    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:18:59.422105 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:18:59.422070    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"]
Jul 24 18:19:00.341025 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:00.340988    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"]
Jul 24 18:19:04.263611 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:04.263576    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:19:11.042215 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.042177    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"]
Jul 24 18:19:11.042840 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.042465    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544" podUID="bf0a346c-1682-4809-9c28-35885ff70ccc" containerName="maas-api" containerID="cri-o://72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9" gracePeriod=30
Jul 24 18:19:11.113409 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.113357    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"]
Jul 24 18:19:11.113675 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.113619    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z" podUID="14242431-a3f2-4bba-9433-850e73872f3c" containerName="payload-processing" containerID="cri-o://ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7" gracePeriod=30
Jul 24 18:19:11.130267 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.130232    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"]
Jul 24 18:19:11.130487 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.130464    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc" podUID="19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" containerName="payload-pre-processing" containerID="cri-o://ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032" gracePeriod=30
Jul 24 18:19:11.389447 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.389419    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:19:11.435453 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.435429    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:19:11.439650 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.439627    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:19:11.446274 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.446252    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5j6b\" (UniqueName: \"kubernetes.io/projected/14242431-a3f2-4bba-9433-850e73872f3c-kube-api-access-z5j6b\") pod \"14242431-a3f2-4bba-9433-850e73872f3c\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") "
Jul 24 18:19:11.446393 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.446342    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14242431-a3f2-4bba-9433-850e73872f3c-plugins-config-volume\") pod \"14242431-a3f2-4bba-9433-850e73872f3c\" (UID: \"14242431-a3f2-4bba-9433-850e73872f3c\") "
Jul 24 18:19:11.446764 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.446730    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14242431-a3f2-4bba-9433-850e73872f3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14242431-a3f2-4bba-9433-850e73872f3c" (UID: "14242431-a3f2-4bba-9433-850e73872f3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:11.448397 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.448354    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14242431-a3f2-4bba-9433-850e73872f3c-kube-api-access-z5j6b" (OuterVolumeSpecName: "kube-api-access-z5j6b") pod "14242431-a3f2-4bba-9433-850e73872f3c" (UID: "14242431-a3f2-4bba-9433-850e73872f3c"). InnerVolumeSpecName "kube-api-access-z5j6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:11.547627 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.547546    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls\") pod \"bf0a346c-1682-4809-9c28-35885ff70ccc\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") "
Jul 24 18:19:11.547785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.547635    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-plugins-config-volume\") pod \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") "
Jul 24 18:19:11.547785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.547662    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrhq8\" (UniqueName: \"kubernetes.io/projected/bf0a346c-1682-4809-9c28-35885ff70ccc-kube-api-access-mrhq8\") pod \"bf0a346c-1682-4809-9c28-35885ff70ccc\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") "
Jul 24 18:19:11.547785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.547681    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7npzz\" (UniqueName: \"kubernetes.io/projected/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-kube-api-access-7npzz\") pod \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\" (UID: \"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88\") "
Jul 24 18:19:11.547785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.547731    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0a346c-1682-4809-9c28-35885ff70ccc-openshift-service-ca\") pod \"bf0a346c-1682-4809-9c28-35885ff70ccc\" (UID: \"bf0a346c-1682-4809-9c28-35885ff70ccc\") "
Jul 24 18:19:11.548101 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.548076    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5j6b\" (UniqueName: \"kubernetes.io/projected/14242431-a3f2-4bba-9433-850e73872f3c-kube-api-access-z5j6b\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:11.548223 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.548107    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14242431-a3f2-4bba-9433-850e73872f3c-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:11.548223 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.548069    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" (UID: "19bc03a7-a6a3-48b3-a600-09d0ca1e5a88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:11.548223 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.548130    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0a346c-1682-4809-9c28-35885ff70ccc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf0a346c-1682-4809-9c28-35885ff70ccc" (UID: "bf0a346c-1682-4809-9c28-35885ff70ccc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:11.549965 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.549937    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf0a346c-1682-4809-9c28-35885ff70ccc" (UID: "bf0a346c-1682-4809-9c28-35885ff70ccc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:19:11.550075 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.549991    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0a346c-1682-4809-9c28-35885ff70ccc-kube-api-access-mrhq8" (OuterVolumeSpecName: "kube-api-access-mrhq8") pod "bf0a346c-1682-4809-9c28-35885ff70ccc" (UID: "bf0a346c-1682-4809-9c28-35885ff70ccc"). InnerVolumeSpecName "kube-api-access-mrhq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:11.550075 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.550044    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-kube-api-access-7npzz" (OuterVolumeSpecName: "kube-api-access-7npzz") pod "19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" (UID: "19bc03a7-a6a3-48b3-a600-09d0ca1e5a88"). InnerVolumeSpecName "kube-api-access-7npzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:11.563509 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.563472    2551 generic.go:356] "Generic (PLEG): container finished" podID="19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" containerID="ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032" exitCode=0
Jul 24 18:19:11.563638 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.563598    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc" event={"ID":"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88","Type":"ContainerDied","Data":"ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032"}
Jul 24 18:19:11.563959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.563685    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc" event={"ID":"19bc03a7-a6a3-48b3-a600-09d0ca1e5a88","Type":"ContainerDied","Data":"c5237ba71ee1078fb8421a38c4e5ec7c0c432b258d9b83643f72cfcc1003318a"}
Jul 24 18:19:11.563959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.563726    2551 scope.go:117] "RemoveContainer" containerID="ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032"
Jul 24 18:19:11.563959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.563925    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"
Jul 24 18:19:11.566986 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.566959    2551 generic.go:356] "Generic (PLEG): container finished" podID="bf0a346c-1682-4809-9c28-35885ff70ccc" containerID="72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9" exitCode=0
Jul 24 18:19:11.567245 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.567225    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544" event={"ID":"bf0a346c-1682-4809-9c28-35885ff70ccc","Type":"ContainerDied","Data":"72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9"}
Jul 24 18:19:11.567405 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.567367    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544" event={"ID":"bf0a346c-1682-4809-9c28-35885ff70ccc","Type":"ContainerDied","Data":"f3773db50394eb9cba9c204923be6f04b7b1cd551fe2d1c874823ef8b8ec451d"}
Jul 24 18:19:11.567902 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.567859    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"
Jul 24 18:19:11.570831 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.570806    2551 generic.go:356] "Generic (PLEG): container finished" podID="14242431-a3f2-4bba-9433-850e73872f3c" containerID="ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7" exitCode=0
Jul 24 18:19:11.570942 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.570870    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"
Jul 24 18:19:11.571001 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.570877    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z" event={"ID":"14242431-a3f2-4bba-9433-850e73872f3c","Type":"ContainerDied","Data":"ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7"}
Jul 24 18:19:11.571001 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.570990    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z" event={"ID":"14242431-a3f2-4bba-9433-850e73872f3c","Type":"ContainerDied","Data":"0bdb9fd0f926a20a83ef47e371abdba4d4a362c6e4d3fb76214b5fcd8b727f5d"}
Jul 24 18:19:11.577180 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.577163    2551 scope.go:117] "RemoveContainer" containerID="ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032"
Jul 24 18:19:11.577465 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:11.577446    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032\": container with ID starting with ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032 not found: ID does not exist" containerID="ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032"
Jul 24 18:19:11.577542 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.577473    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032"} err="failed to get container status \"ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032\": rpc error: code = NotFound desc = could not find container \"ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032\": container with ID starting with ae93a7a5c0ec20f92ff3eb94fe4972b1da13beae9cc7abe0570b6604175d7032 not found: ID does not exist"
Jul 24 18:19:11.577542 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.577490    2551 scope.go:117] "RemoveContainer" containerID="72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9"
Jul 24 18:19:11.587061 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.587041    2551 scope.go:117] "RemoveContainer" containerID="72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9"
Jul 24 18:19:11.590825 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:11.590795    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9\": container with ID starting with 72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9 not found: ID does not exist" containerID="72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9"
Jul 24 18:19:11.590908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.590838    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9"} err="failed to get container status \"72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9\": rpc error: code = NotFound desc = could not find container \"72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9\": container with ID starting with 72dcbbf0b0d05f2e266648efd431c0e4430ff6eb1f5d59085a050fbc271d95a9 not found: ID does not exist"
Jul 24 18:19:11.590908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.590870    2551 scope.go:117] "RemoveContainer" containerID="ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7"
Jul 24 18:19:11.598624 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.598598    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"]
Jul 24 18:19:11.600700 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.600676    2551 scope.go:117] "RemoveContainer" containerID="ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7"
Jul 24 18:19:11.600976 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:11.600956    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7\": container with ID starting with ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7 not found: ID does not exist" containerID="ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7"
Jul 24 18:19:11.601045 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.600987    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7"} err="failed to get container status \"ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7\": rpc error: code = NotFound desc = could not find container \"ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7\": container with ID starting with ca00b454f0c9f09a2abc41998206c413e930040bdb8ada4d21d6a3934c6267b7 not found: ID does not exist"
Jul 24 18:19:11.604980 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.604960    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e3964-7f4988cc6d-nrgcc"]
Jul 24 18:19:11.617469 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.617442    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"]
Jul 24 18:19:11.625818 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.625798    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e3964-5cd85fbb77-7m544"]
Jul 24 18:19:11.638129 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.638101    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"]
Jul 24 18:19:11.645397 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.645363    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e3964-5b686484b8-g769z"]
Jul 24 18:19:11.649076 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.649054    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:11.649165 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.649079    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrhq8\" (UniqueName: \"kubernetes.io/projected/bf0a346c-1682-4809-9c28-35885ff70ccc-kube-api-access-mrhq8\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:11.649165 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.649095    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npzz\" (UniqueName: \"kubernetes.io/projected/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88-kube-api-access-7npzz\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:11.649165 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.649108    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0a346c-1682-4809-9c28-35885ff70ccc-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:11.649165 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:11.649121    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf0a346c-1682-4809-9c28-35885ff70ccc-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:13.301100 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:13.301068    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14242431-a3f2-4bba-9433-850e73872f3c" path="/var/lib/kubelet/pods/14242431-a3f2-4bba-9433-850e73872f3c/volumes"
Jul 24 18:19:13.301493 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:13.301446    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" path="/var/lib/kubelet/pods/19bc03a7-a6a3-48b3-a600-09d0ca1e5a88/volumes"
Jul 24 18:19:13.301803 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:13.301788    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0a346c-1682-4809-9c28-35885ff70ccc" path="/var/lib/kubelet/pods/bf0a346c-1682-4809-9c28-35885ff70ccc/volumes"
Jul 24 18:19:14.442257 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.442210    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j" podUID="5f058d4f-555d-476c-b4b9-84d4894c307c" containerName="main" containerID="cri-o://74e9ca8968bfeaab4a4b30ab2892473802520eb4811b76505aec322d4b58d634" gracePeriod=45
Jul 24 18:19:14.591570 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.591530    2551 generic.go:356] "Generic (PLEG): container finished" podID="5f058d4f-555d-476c-b4b9-84d4894c307c" containerID="74e9ca8968bfeaab4a4b30ab2892473802520eb4811b76505aec322d4b58d634" exitCode=0
Jul 24 18:19:14.591734 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.591598    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j" event={"ID":"5f058d4f-555d-476c-b4b9-84d4894c307c","Type":"ContainerDied","Data":"74e9ca8968bfeaab4a4b30ab2892473802520eb4811b76505aec322d4b58d634"}
Jul 24 18:19:14.701471 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.701405    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:19:14.777255 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777216    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f058d4f-555d-476c-b4b9-84d4894c307c-tls-certs\") pod \"5f058d4f-555d-476c-b4b9-84d4894c307c\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") "
Jul 24 18:19:14.777473 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777267    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-home\") pod \"5f058d4f-555d-476c-b4b9-84d4894c307c\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") "
Jul 24 18:19:14.777473 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777307    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-model-cache\") pod \"5f058d4f-555d-476c-b4b9-84d4894c307c\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") "
Jul 24 18:19:14.777473 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777347    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-dshm\") pod \"5f058d4f-555d-476c-b4b9-84d4894c307c\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") "
Jul 24 18:19:14.777473 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777403    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-tmp-dir\") pod \"5f058d4f-555d-476c-b4b9-84d4894c307c\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") "
Jul 24 18:19:14.777473 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777449    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rljqp\" (UniqueName: \"kubernetes.io/projected/5f058d4f-555d-476c-b4b9-84d4894c307c-kube-api-access-rljqp\") pod \"5f058d4f-555d-476c-b4b9-84d4894c307c\" (UID: \"5f058d4f-555d-476c-b4b9-84d4894c307c\") "
Jul 24 18:19:14.777749 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777613    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f058d4f-555d-476c-b4b9-84d4894c307c" (UID: "5f058d4f-555d-476c-b4b9-84d4894c307c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.777749 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777628    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-home" (OuterVolumeSpecName: "home") pod "5f058d4f-555d-476c-b4b9-84d4894c307c" (UID: "5f058d4f-555d-476c-b4b9-84d4894c307c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.777749 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777681    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f058d4f-555d-476c-b4b9-84d4894c307c" (UID: "5f058d4f-555d-476c-b4b9-84d4894c307c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.777916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777877    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.777916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777893    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.777916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.777904    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.779799 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.779772    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f058d4f-555d-476c-b4b9-84d4894c307c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f058d4f-555d-476c-b4b9-84d4894c307c" (UID: "5f058d4f-555d-476c-b4b9-84d4894c307c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:19:14.779799 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.779786    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-dshm" (OuterVolumeSpecName: "dshm") pod "5f058d4f-555d-476c-b4b9-84d4894c307c" (UID: "5f058d4f-555d-476c-b4b9-84d4894c307c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:14.779799 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.779788    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f058d4f-555d-476c-b4b9-84d4894c307c-kube-api-access-rljqp" (OuterVolumeSpecName: "kube-api-access-rljqp") pod "5f058d4f-555d-476c-b4b9-84d4894c307c" (UID: "5f058d4f-555d-476c-b4b9-84d4894c307c"). InnerVolumeSpecName "kube-api-access-rljqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:14.879250 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.879213    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rljqp\" (UniqueName: \"kubernetes.io/projected/5f058d4f-555d-476c-b4b9-84d4894c307c-kube-api-access-rljqp\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.879250 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.879245    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f058d4f-555d-476c-b4b9-84d4894c307c-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:14.879250 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:14.879256    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f058d4f-555d-476c-b4b9-84d4894c307c-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:15.361675 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.361631    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk" podUID="d1039a87-e1eb-4e89-81a9-0f9acb4c4141" containerName="main" containerID="cri-o://96a8d342098c88d8a113cb7cf58aea91ec7a4f80d51206ff258d0f1e7545d183" gracePeriod=45
Jul 24 18:19:15.599022 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.598940    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"
Jul 24 18:19:15.599022 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.598954    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j" event={"ID":"5f058d4f-555d-476c-b4b9-84d4894c307c","Type":"ContainerDied","Data":"f7ff62fafa8e4d6bc60b3ae203bae94f52806f1eb8a780582b29c6673149e634"}
Jul 24 18:19:15.599022 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.599014    2551 scope.go:117] "RemoveContainer" containerID="74e9ca8968bfeaab4a4b30ab2892473802520eb4811b76505aec322d4b58d634"
Jul 24 18:19:15.601846 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.601819    2551 generic.go:356] "Generic (PLEG): container finished" podID="d1039a87-e1eb-4e89-81a9-0f9acb4c4141" containerID="96a8d342098c88d8a113cb7cf58aea91ec7a4f80d51206ff258d0f1e7545d183" exitCode=0
Jul 24 18:19:15.601846 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.601852    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk" event={"ID":"d1039a87-e1eb-4e89-81a9-0f9acb4c4141","Type":"ContainerDied","Data":"96a8d342098c88d8a113cb7cf58aea91ec7a4f80d51206ff258d0f1e7545d183"}
Jul 24 18:19:15.623899 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.623877    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:19:15.634074 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.634049    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"]
Jul 24 18:19:15.638233 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.638208    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6e3964/test-model-6e3964-kserve-86575cc7c9-k726j"]
Jul 24 18:19:15.787097 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787064    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tls-certs\") pod \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") "
Jul 24 18:19:15.787302 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787114    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-home\") pod \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") "
Jul 24 18:19:15.787302 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787144    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-model-cache\") pod \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") "
Jul 24 18:19:15.787302 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787185    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-dshm\") pod \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") "
Jul 24 18:19:15.787302 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787213    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtwjq\" (UniqueName: \"kubernetes.io/projected/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-kube-api-access-xtwjq\") pod \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") "
Jul 24 18:19:15.787302 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787240    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tmp-dir\") pod \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\" (UID: \"d1039a87-e1eb-4e89-81a9-0f9acb4c4141\") "
Jul 24 18:19:15.787593 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787500    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-home" (OuterVolumeSpecName: "home") pod "d1039a87-e1eb-4e89-81a9-0f9acb4c4141" (UID: "d1039a87-e1eb-4e89-81a9-0f9acb4c4141"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:15.787593 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787512    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1039a87-e1eb-4e89-81a9-0f9acb4c4141" (UID: "d1039a87-e1eb-4e89-81a9-0f9acb4c4141"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:15.787743 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787723    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1039a87-e1eb-4e89-81a9-0f9acb4c4141" (UID: "d1039a87-e1eb-4e89-81a9-0f9acb4c4141"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:15.787807 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787729    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:15.787807 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.787771    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:15.789658 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.789632    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-dshm" (OuterVolumeSpecName: "dshm") pod "d1039a87-e1eb-4e89-81a9-0f9acb4c4141" (UID: "d1039a87-e1eb-4e89-81a9-0f9acb4c4141"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:19:15.789863 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.789841    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-kube-api-access-xtwjq" (OuterVolumeSpecName: "kube-api-access-xtwjq") pod "d1039a87-e1eb-4e89-81a9-0f9acb4c4141" (UID: "d1039a87-e1eb-4e89-81a9-0f9acb4c4141"). InnerVolumeSpecName "kube-api-access-xtwjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:15.789947 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.789870    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1039a87-e1eb-4e89-81a9-0f9acb4c4141" (UID: "d1039a87-e1eb-4e89-81a9-0f9acb4c4141"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:19:15.888901 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.888799    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:15.888901 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.888842    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:15.888901 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.888856    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwjq\" (UniqueName: \"kubernetes.io/projected/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-kube-api-access-xtwjq\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:15.888901 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:15.888869    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1039a87-e1eb-4e89-81a9-0f9acb4c4141-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:16.609289 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:16.609255    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"
Jul 24 18:19:16.609898 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:16.609300    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk" event={"ID":"d1039a87-e1eb-4e89-81a9-0f9acb4c4141","Type":"ContainerDied","Data":"90d2dc7e02c928328ce3545d7e9f60997d7f8eddabbb2cefb7660da282d7024b"}
Jul 24 18:19:16.609898 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:16.609353    2551 scope.go:117] "RemoveContainer" containerID="96a8d342098c88d8a113cb7cf58aea91ec7a4f80d51206ff258d0f1e7545d183"
Jul 24 18:19:16.643494 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:16.643459    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"]
Jul 24 18:19:16.650447 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:16.650424    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0f1d5b/test-model-0f1d5b-kserve-7475677dc7-tvhgk"]
Jul 24 18:19:17.300112 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:17.300073    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f058d4f-555d-476c-b4b9-84d4894c307c" path="/var/lib/kubelet/pods/5f058d4f-555d-476c-b4b9-84d4894c307c/volumes"
Jul 24 18:19:17.300599 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:17.300577    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1039a87-e1eb-4e89-81a9-0f9acb4c4141" path="/var/lib/kubelet/pods/d1039a87-e1eb-4e89-81a9-0f9acb4c4141/volumes"
Jul 24 18:19:19.955854 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:19.955820    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:19:33.119598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.119560    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"]
Jul 24 18:19:33.120443 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.120410    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs" podUID="c1a46397-2514-46ac-b94a-f0fa18d9d717" containerName="maas-api" containerID="cri-o://f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc" gracePeriod=30
Jul 24 18:19:33.196415 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.196355    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"]
Jul 24 18:19:33.196662 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.196638    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76" podUID="16664846-4bc9-4e22-befe-11ccd41f0d7d" containerName="payload-processing" containerID="cri-o://bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325" gracePeriod=30
Jul 24 18:19:33.204328 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.204183    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"]
Jul 24 18:19:33.204502 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.204442    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns" podUID="ac7d2ea2-9c43-438d-a360-35b99d527349" containerName="payload-pre-processing" containerID="cri-o://93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b" gracePeriod=30
Jul 24 18:19:33.469703 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.469676    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:19:33.480004 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.479978    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:19:33.500815 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.500783    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:19:33.567433 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567401    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a46397-2514-46ac-b94a-f0fa18d9d717-openshift-service-ca\") pod \"c1a46397-2514-46ac-b94a-f0fa18d9d717\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") "
Jul 24 18:19:33.567589 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567474    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7d2ea2-9c43-438d-a360-35b99d527349-plugins-config-volume\") pod \"ac7d2ea2-9c43-438d-a360-35b99d527349\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") "
Jul 24 18:19:33.567589 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567513    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnds8\" (UniqueName: \"kubernetes.io/projected/16664846-4bc9-4e22-befe-11ccd41f0d7d-kube-api-access-dnds8\") pod \"16664846-4bc9-4e22-befe-11ccd41f0d7d\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") "
Jul 24 18:19:33.567589 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567528    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16664846-4bc9-4e22-befe-11ccd41f0d7d-plugins-config-volume\") pod \"16664846-4bc9-4e22-befe-11ccd41f0d7d\" (UID: \"16664846-4bc9-4e22-befe-11ccd41f0d7d\") "
Jul 24 18:19:33.567589 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567550    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xbfn\" (UniqueName: \"kubernetes.io/projected/c1a46397-2514-46ac-b94a-f0fa18d9d717-kube-api-access-9xbfn\") pod \"c1a46397-2514-46ac-b94a-f0fa18d9d717\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") "
Jul 24 18:19:33.567589 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567569    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls\") pod \"c1a46397-2514-46ac-b94a-f0fa18d9d717\" (UID: \"c1a46397-2514-46ac-b94a-f0fa18d9d717\") "
Jul 24 18:19:33.567846 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567614    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mntnm\" (UniqueName: \"kubernetes.io/projected/ac7d2ea2-9c43-438d-a360-35b99d527349-kube-api-access-mntnm\") pod \"ac7d2ea2-9c43-438d-a360-35b99d527349\" (UID: \"ac7d2ea2-9c43-438d-a360-35b99d527349\") "
Jul 24 18:19:33.567846 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567708    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a46397-2514-46ac-b94a-f0fa18d9d717-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1a46397-2514-46ac-b94a-f0fa18d9d717" (UID: "c1a46397-2514-46ac-b94a-f0fa18d9d717"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:33.567954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567905    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7d2ea2-9c43-438d-a360-35b99d527349-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac7d2ea2-9c43-438d-a360-35b99d527349" (UID: "ac7d2ea2-9c43-438d-a360-35b99d527349"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:33.567954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567916    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a46397-2514-46ac-b94a-f0fa18d9d717-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:33.568066 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.567956    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16664846-4bc9-4e22-befe-11ccd41f0d7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16664846-4bc9-4e22-befe-11ccd41f0d7d" (UID: "16664846-4bc9-4e22-befe-11ccd41f0d7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:19:33.569943 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.569906    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a46397-2514-46ac-b94a-f0fa18d9d717-kube-api-access-9xbfn" (OuterVolumeSpecName: "kube-api-access-9xbfn") pod "c1a46397-2514-46ac-b94a-f0fa18d9d717" (UID: "c1a46397-2514-46ac-b94a-f0fa18d9d717"). InnerVolumeSpecName "kube-api-access-9xbfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:33.570057 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.569957    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1a46397-2514-46ac-b94a-f0fa18d9d717" (UID: "c1a46397-2514-46ac-b94a-f0fa18d9d717"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:19:33.570294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.570274    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16664846-4bc9-4e22-befe-11ccd41f0d7d-kube-api-access-dnds8" (OuterVolumeSpecName: "kube-api-access-dnds8") pod "16664846-4bc9-4e22-befe-11ccd41f0d7d" (UID: "16664846-4bc9-4e22-befe-11ccd41f0d7d"). InnerVolumeSpecName "kube-api-access-dnds8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:33.570294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.570281    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7d2ea2-9c43-438d-a360-35b99d527349-kube-api-access-mntnm" (OuterVolumeSpecName: "kube-api-access-mntnm") pod "ac7d2ea2-9c43-438d-a360-35b99d527349" (UID: "ac7d2ea2-9c43-438d-a360-35b99d527349"). InnerVolumeSpecName "kube-api-access-mntnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:19:33.668745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.668666    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbfn\" (UniqueName: \"kubernetes.io/projected/c1a46397-2514-46ac-b94a-f0fa18d9d717-kube-api-access-9xbfn\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:33.668745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.668692    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a46397-2514-46ac-b94a-f0fa18d9d717-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:33.668745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.668702    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mntnm\" (UniqueName: \"kubernetes.io/projected/ac7d2ea2-9c43-438d-a360-35b99d527349-kube-api-access-mntnm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:33.668745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.668710    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7d2ea2-9c43-438d-a360-35b99d527349-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:33.668745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.668719    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnds8\" (UniqueName: \"kubernetes.io/projected/16664846-4bc9-4e22-befe-11ccd41f0d7d-kube-api-access-dnds8\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:33.668745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.668728    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16664846-4bc9-4e22-befe-11ccd41f0d7d-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:19:33.706360 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.706308    2551 generic.go:356] "Generic (PLEG): container finished" podID="c1a46397-2514-46ac-b94a-f0fa18d9d717" containerID="f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc" exitCode=0
Jul 24 18:19:33.706360 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.706360    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"
Jul 24 18:19:33.706733 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.706406    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs" event={"ID":"c1a46397-2514-46ac-b94a-f0fa18d9d717","Type":"ContainerDied","Data":"f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc"}
Jul 24 18:19:33.706733 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.706469    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs" event={"ID":"c1a46397-2514-46ac-b94a-f0fa18d9d717","Type":"ContainerDied","Data":"c2283356f01d0edc7a5774a07643a6b96f8b380803584734fe6a0c4fc1e6be45"}
Jul 24 18:19:33.706733 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.706500    2551 scope.go:117] "RemoveContainer" containerID="f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc"
Jul 24 18:19:33.708044 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.708022    2551 generic.go:356] "Generic (PLEG): container finished" podID="16664846-4bc9-4e22-befe-11ccd41f0d7d" containerID="bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325" exitCode=0
Jul 24 18:19:33.708144 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.708113    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76" event={"ID":"16664846-4bc9-4e22-befe-11ccd41f0d7d","Type":"ContainerDied","Data":"bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325"}
Jul 24 18:19:33.708205 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.708156    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76" event={"ID":"16664846-4bc9-4e22-befe-11ccd41f0d7d","Type":"ContainerDied","Data":"883b86f8b3aa6dfb430a40517fe572c8a37e0bfea391d4a236464cbcab14d172"}
Jul 24 18:19:33.708263 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.708222    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"
Jul 24 18:19:33.710108 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.710073    2551 generic.go:356] "Generic (PLEG): container finished" podID="ac7d2ea2-9c43-438d-a360-35b99d527349" containerID="93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b" exitCode=0
Jul 24 18:19:33.710224 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.710123    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"
Jul 24 18:19:33.710224 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.710129    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns" event={"ID":"ac7d2ea2-9c43-438d-a360-35b99d527349","Type":"ContainerDied","Data":"93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b"}
Jul 24 18:19:33.710342 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.710240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns" event={"ID":"ac7d2ea2-9c43-438d-a360-35b99d527349","Type":"ContainerDied","Data":"5f8a7c9a446b986773a56e8dc8e11cdf5ce9ce2c8d1c84f5c332a5684f1f0dbb"}
Jul 24 18:19:33.718310 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.718283    2551 scope.go:117] "RemoveContainer" containerID="f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc"
Jul 24 18:19:33.720882 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:33.720847    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc\": container with ID starting with f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc not found: ID does not exist" containerID="f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc"
Jul 24 18:19:33.721000 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.720891    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc"} err="failed to get container status \"f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc\": rpc error: code = NotFound desc = could not find container \"f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc\": container with ID starting with f6b02a9f80398d08790351b8ca62e9ef49207220755605b53d7437663d676dbc not found: ID does not exist"
Jul 24 18:19:33.721000 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.720957    2551 scope.go:117] "RemoveContainer" containerID="bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325"
Jul 24 18:19:33.735160 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.735140    2551 scope.go:117] "RemoveContainer" containerID="bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325"
Jul 24 18:19:33.735453 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:33.735431    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325\": container with ID starting with bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325 not found: ID does not exist" containerID="bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325"
Jul 24 18:19:33.735542 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.735466    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325"} err="failed to get container status \"bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325\": rpc error: code = NotFound desc = could not find container \"bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325\": container with ID starting with bb6b609424300cf1420a7037c5fd20cea1144057080614d66aa5138af2b2f325 not found: ID does not exist"
Jul 24 18:19:33.735542 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.735489    2551 scope.go:117] "RemoveContainer" containerID="93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b"
Jul 24 18:19:33.739868 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.739847    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"]
Jul 24 18:19:33.744430 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.744397    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0f1d5b-84d6b7f785-jz5cs"]
Jul 24 18:19:33.744492 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.744474    2551 scope.go:117] "RemoveContainer" containerID="93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b"
Jul 24 18:19:33.744768 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:33.744744    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b\": container with ID starting with 93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b not found: ID does not exist" containerID="93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b"
Jul 24 18:19:33.744841 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.744789    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b"} err="failed to get container status \"93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b\": rpc error: code = NotFound desc = could not find container \"93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b\": container with ID starting with 93dffd4a2005d6190b5ca37bad631847d9f9209b7a374c400d1195a6748dd91b not found: ID does not exist"
Jul 24 18:19:33.754279 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.754250    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"]
Jul 24 18:19:33.758443 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.758422    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0f1d5b-59f9b576d-f2b76"]
Jul 24 18:19:33.768146 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.768115    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"]
Jul 24 18:19:33.770344 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:33.770322    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0f1d5b-74b8f7d85b-d6vns"]
Jul 24 18:19:35.300433 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:35.300401    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16664846-4bc9-4e22-befe-11ccd41f0d7d" path="/var/lib/kubelet/pods/16664846-4bc9-4e22-befe-11ccd41f0d7d/volumes"
Jul 24 18:19:35.300807 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:35.300763    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac7d2ea2-9c43-438d-a360-35b99d527349" path="/var/lib/kubelet/pods/ac7d2ea2-9c43-438d-a360-35b99d527349/volumes"
Jul 24 18:19:35.301090 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:35.301077    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a46397-2514-46ac-b94a-f0fa18d9d717" path="/var/lib/kubelet/pods/c1a46397-2514-46ac-b94a-f0fa18d9d717/volumes"
Jul 24 18:19:35.955596 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:35.955561    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:19:48.603252 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603208    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"]
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603602    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14242431-a3f2-4bba-9433-850e73872f3c" containerName="payload-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603613    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="14242431-a3f2-4bba-9433-850e73872f3c" containerName="payload-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603622    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16664846-4bc9-4e22-befe-11ccd41f0d7d" containerName="payload-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603627    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="16664846-4bc9-4e22-befe-11ccd41f0d7d" containerName="payload-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603639    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a46397-2514-46ac-b94a-f0fa18d9d717" containerName="maas-api"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603644    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a46397-2514-46ac-b94a-f0fa18d9d717" containerName="maas-api"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603651    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac7d2ea2-9c43-438d-a360-35b99d527349" containerName="payload-pre-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603656    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac7d2ea2-9c43-438d-a360-35b99d527349" containerName="payload-pre-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603663    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1039a87-e1eb-4e89-81a9-0f9acb4c4141" containerName="main"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603668    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1039a87-e1eb-4e89-81a9-0f9acb4c4141" containerName="main"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603675    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f058d4f-555d-476c-b4b9-84d4894c307c" containerName="main"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603680    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f058d4f-555d-476c-b4b9-84d4894c307c" containerName="main"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603687    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" containerName="payload-pre-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603692    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" containerName="payload-pre-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603704    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0a346c-1682-4809-9c28-35885ff70ccc" containerName="maas-api"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603711    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0a346c-1682-4809-9c28-35885ff70ccc" containerName="maas-api"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603759    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="16664846-4bc9-4e22-befe-11ccd41f0d7d" containerName="payload-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603769    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bc03a7-a6a3-48b3-a600-09d0ca1e5a88" containerName="payload-pre-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603775    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0a346c-1682-4809-9c28-35885ff70ccc" containerName="maas-api"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603781    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f058d4f-555d-476c-b4b9-84d4894c307c" containerName="main"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603787    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a46397-2514-46ac-b94a-f0fa18d9d717" containerName="maas-api"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603793    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="14242431-a3f2-4bba-9433-850e73872f3c" containerName="payload-processing"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603798    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1039a87-e1eb-4e89-81a9-0f9acb4c4141" containerName="main"
Jul 24 18:19:48.603924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.603802    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac7d2ea2-9c43-438d-a360-35b99d527349" containerName="payload-pre-processing"
Jul 24 18:19:48.606957 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.606937    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.610055 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.610029    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:19:48.610188 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.610057    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:19:48.610756 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.610741    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-013528-serving-cert\""
Jul 24 18:19:48.610814 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.610762    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:19:48.616801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.616778    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"]
Jul 24 18:19:48.715547 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.715517    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qln\" (UniqueName: \"kubernetes.io/projected/dbbaefc9-56ec-4812-a395-f3e169710a12-kube-api-access-q5qln\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.715723 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.715572    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbbaefc9-56ec-4812-a395-f3e169710a12-maas-api-tls\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.715723 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.715685    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbaefc9-56ec-4812-a395-f3e169710a12-openshift-service-ca\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.817226 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.817191    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qln\" (UniqueName: \"kubernetes.io/projected/dbbaefc9-56ec-4812-a395-f3e169710a12-kube-api-access-q5qln\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.817445 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.817239    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbbaefc9-56ec-4812-a395-f3e169710a12-maas-api-tls\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.817445 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.817293    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbaefc9-56ec-4812-a395-f3e169710a12-openshift-service-ca\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.818058 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.818039    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbaefc9-56ec-4812-a395-f3e169710a12-openshift-service-ca\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.819883 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.819863    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbbaefc9-56ec-4812-a395-f3e169710a12-maas-api-tls\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.827698 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.827677    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qln\" (UniqueName: \"kubernetes.io/projected/dbbaefc9-56ec-4812-a395-f3e169710a12-kube-api-access-q5qln\") pod \"maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:48.918576 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:48.918485    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:49.059274 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.059242    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"]
Jul 24 18:19:49.060182 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:19:49.060155    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbaefc9_56ec_4812_a395_f3e169710a12.slice/crio-5f2b684ef96528d6251a497a36ea4d486c0e8ac5be371135878dc1525dcee6ea WatchSource:0}: Error finding container 5f2b684ef96528d6251a497a36ea4d486c0e8ac5be371135878dc1525dcee6ea: Status 404 returned error can't find the container with id 5f2b684ef96528d6251a497a36ea4d486c0e8ac5be371135878dc1525dcee6ea
Jul 24 18:19:49.061881 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.061862    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:19:49.595117 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.595091    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"]
Jul 24 18:19:49.600081 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.600059    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.603962 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.603940    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-013528\""
Jul 24 18:19:49.605461 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.605440    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-013528-dockercfg-qp9r5\""
Jul 24 18:19:49.614413 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.614392    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"]
Jul 24 18:19:49.676990 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.676959    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"]
Jul 24 18:19:49.680471 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.680449    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:49.691306 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.691242    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"]
Jul 24 18:19:49.726338 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.726303    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf7mj\" (UniqueName: \"kubernetes.io/projected/00a84d83-2bce-4440-afa9-696169897e40-kube-api-access-bf7mj\") pod \"payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.726520 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.726356    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a84d83-2bce-4440-afa9-696169897e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.794061 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.794025    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg" event={"ID":"dbbaefc9-56ec-4812-a395-f3e169710a12","Type":"ContainerStarted","Data":"5bda53ff80d709604e7a830f55fe2dd74bb675fdc0dd0d375454967048b10a79"}
Jul 24 18:19:49.794254 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.794088    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg" event={"ID":"dbbaefc9-56ec-4812-a395-f3e169710a12","Type":"ContainerStarted","Data":"5f2b684ef96528d6251a497a36ea4d486c0e8ac5be371135878dc1525dcee6ea"}
Jul 24 18:19:49.794321 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.794295    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:49.815186 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.815083    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg" podStartSLOduration=1.32189171 podStartE2EDuration="1.815066529s" totalImagesPullingTime="493.174819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:49.812570525 +0000 UTC m=+2380.929497865" watchObservedRunningTime="2026-07-24 18:19:49.815066529 +0000 UTC m=+2380.931993868"
Jul 24 18:19:49.827831 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.827802    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf7mj\" (UniqueName: \"kubernetes.io/projected/00a84d83-2bce-4440-afa9-696169897e40-kube-api-access-bf7mj\") pod \"payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.827974 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.827844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a84d83-2bce-4440-afa9-696169897e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.827974 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.827909    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f1fe51-94b6-4894-9c4b-4798a7693f82-plugins-config-volume\") pod \"payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") " pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:49.827974 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.827939    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfns4\" (UniqueName: \"kubernetes.io/projected/c0f1fe51-94b6-4894-9c4b-4798a7693f82-kube-api-access-xfns4\") pod \"payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") " pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:49.828679 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.828662    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a84d83-2bce-4440-afa9-696169897e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.838252 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.838220    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf7mj\" (UniqueName: \"kubernetes.io/projected/00a84d83-2bce-4440-afa9-696169897e40-kube-api-access-bf7mj\") pod \"payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.909921 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.909872    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:49.929395 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.929347    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f1fe51-94b6-4894-9c4b-4798a7693f82-plugins-config-volume\") pod \"payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") " pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:49.929706 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.929675    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfns4\" (UniqueName: \"kubernetes.io/projected/c0f1fe51-94b6-4894-9c4b-4798a7693f82-kube-api-access-xfns4\") pod \"payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") " pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:49.930306 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.930279    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f1fe51-94b6-4894-9c4b-4798a7693f82-plugins-config-volume\") pod \"payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") " pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:49.938729 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.938703    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfns4\" (UniqueName: \"kubernetes.io/projected/c0f1fe51-94b6-4894-9c4b-4798a7693f82-kube-api-access-xfns4\") pod \"payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") " pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:49.994401 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:49.994098    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:50.079341 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.079290    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"]
Jul 24 18:19:50.082701 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:19:50.082551    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a84d83_2bce_4440_afa9_696169897e40.slice/crio-0f13991dd600a193ede29acf235aaae38498d09e3b124822b410ed7eee83071c WatchSource:0}: Error finding container 0f13991dd600a193ede29acf235aaae38498d09e3b124822b410ed7eee83071c: Status 404 returned error can't find the container with id 0f13991dd600a193ede29acf235aaae38498d09e3b124822b410ed7eee83071c
Jul 24 18:19:50.178456 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.178326    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"]
Jul 24 18:19:50.181096 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:19:50.181070    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f1fe51_94b6_4894_9c4b_4798a7693f82.slice/crio-d0836306e231c715b8dbc95cc0fbde4e54fbf1c0ac0b982a7d811610a77e004f WatchSource:0}: Error finding container d0836306e231c715b8dbc95cc0fbde4e54fbf1c0ac0b982a7d811610a77e004f: Status 404 returned error can't find the container with id d0836306e231c715b8dbc95cc0fbde4e54fbf1c0ac0b982a7d811610a77e004f
Jul 24 18:19:50.801926 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.801888    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg" event={"ID":"c0f1fe51-94b6-4894-9c4b-4798a7693f82","Type":"ContainerStarted","Data":"93660daada3b26426d78e1e1c5d894e0f1fc0a2b21f46efc5dba224b4a784699"}
Jul 24 18:19:50.802481 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.801953    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg" event={"ID":"c0f1fe51-94b6-4894-9c4b-4798a7693f82","Type":"ContainerStarted","Data":"d0836306e231c715b8dbc95cc0fbde4e54fbf1c0ac0b982a7d811610a77e004f"}
Jul 24 18:19:50.802481 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.802014    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:19:50.803629 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.803595    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8" event={"ID":"00a84d83-2bce-4440-afa9-696169897e40","Type":"ContainerStarted","Data":"e3ce524ffecda8ce78acf4419998f0f83798fbf42f52dbe8752aec1e9cf58be6"}
Jul 24 18:19:50.803755 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.803644    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8" event={"ID":"00a84d83-2bce-4440-afa9-696169897e40","Type":"ContainerStarted","Data":"0f13991dd600a193ede29acf235aaae38498d09e3b124822b410ed7eee83071c"}
Jul 24 18:19:50.821624 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.821573    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg" podStartSLOduration=1.821557535 podStartE2EDuration="1.821557535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:50.820019516 +0000 UTC m=+2381.936946851" watchObservedRunningTime="2026-07-24 18:19:50.821557535 +0000 UTC m=+2381.938484874"
Jul 24 18:19:50.839791 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:50.839745    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8" podStartSLOduration=1.839730874 podStartE2EDuration="1.839730874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:50.83881691 +0000 UTC m=+2381.955744251" watchObservedRunningTime="2026-07-24 18:19:50.839730874 +0000 UTC m=+2381.956658215"
Jul 24 18:19:51.818594 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:51.818555    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:19:55.355955 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:55.355924    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:19:55.808851 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:55.808824    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:19:56.681787 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.681753    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"]
Jul 24 18:19:56.686406 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.686362    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.688924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.688895    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d1a9df-serving-cert\""
Jul 24 18:19:56.698830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.698806    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"]
Jul 24 18:19:56.796774 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.796731    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.796940 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.796811    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgzk\" (UniqueName: \"kubernetes.io/projected/c8424797-6952-4900-9833-903463a900e4-kube-api-access-9tgzk\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.796940 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.796868    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8424797-6952-4900-9833-903463a900e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.898017 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.897983    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.898195 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.898034    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgzk\" (UniqueName: \"kubernetes.io/projected/c8424797-6952-4900-9833-903463a900e4-kube-api-access-9tgzk\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.898195 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.898072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8424797-6952-4900-9833-903463a900e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.898195 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:56.898148    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-serving-cert: secret "maas-api-e2e-api-b-d1a9df-serving-cert" not found
Jul 24 18:19:56.898308 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:19:56.898228    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls podName:c8424797-6952-4900-9833-903463a900e4 nodeName:}" failed. No retries permitted until 2026-07-24 18:19:57.398208446 +0000 UTC m=+2388.515135767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls") pod "maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7" (UID: "c8424797-6952-4900-9833-903463a900e4") : secret "maas-api-e2e-api-b-d1a9df-serving-cert" not found
Jul 24 18:19:56.898729 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.898713    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8424797-6952-4900-9833-903463a900e4-openshift-service-ca\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:56.908869 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:56.908839    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgzk\" (UniqueName: \"kubernetes.io/projected/c8424797-6952-4900-9833-903463a900e4-kube-api-access-9tgzk\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:57.402932 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.402898    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:57.405767 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.405739    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls\") pod \"maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:57.544563 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.544525    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"]
Jul 24 18:19:57.548475 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.548451    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:57.550877 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.550846    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d1a9df\""
Jul 24 18:19:57.551001 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.550940    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d1a9df-dockercfg-vx6s9\""
Jul 24 18:19:57.565589 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.565559    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"]
Jul 24 18:19:57.598979 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.598953    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:57.706047 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.706013    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t65k\" (UniqueName: \"kubernetes.io/projected/62d82e8d-8ef6-4f4e-8f30-98973140f10c-kube-api-access-4t65k\") pod \"payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:57.706418 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.706255    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d82e8d-8ef6-4f4e-8f30-98973140f10c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:57.743895 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.743743    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"]
Jul 24 18:19:57.746515 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:19:57.746483    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8424797_6952_4900_9833_903463a900e4.slice/crio-14c04f1f5b3ea0d4e884c5760d53e1b62c58480fcd676594116318004e743958 WatchSource:0}: Error finding container 14c04f1f5b3ea0d4e884c5760d53e1b62c58480fcd676594116318004e743958: Status 404 returned error can't find the container with id 14c04f1f5b3ea0d4e884c5760d53e1b62c58480fcd676594116318004e743958
Jul 24 18:19:57.807299 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.807264    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d82e8d-8ef6-4f4e-8f30-98973140f10c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:57.807478 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.807309    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t65k\" (UniqueName: \"kubernetes.io/projected/62d82e8d-8ef6-4f4e-8f30-98973140f10c-kube-api-access-4t65k\") pod \"payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:57.808026 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.808005    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d82e8d-8ef6-4f4e-8f30-98973140f10c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:57.815841 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.815811    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t65k\" (UniqueName: \"kubernetes.io/projected/62d82e8d-8ef6-4f4e-8f30-98973140f10c-kube-api-access-4t65k\") pod \"payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:57.850069 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.850040    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7" event={"ID":"c8424797-6952-4900-9833-903463a900e4","Type":"ContainerStarted","Data":"14c04f1f5b3ea0d4e884c5760d53e1b62c58480fcd676594116318004e743958"}
Jul 24 18:19:57.860360 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:57.860328    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:58.001088 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.001007    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"]
Jul 24 18:19:58.003594 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:19:58.003567    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d82e8d_8ef6_4f4e_8f30_98973140f10c.slice/crio-ff4154c6dbf042fe0f88e6aa873bc0606d60d4b0da953cab6e446dcfc6ccaa3b WatchSource:0}: Error finding container ff4154c6dbf042fe0f88e6aa873bc0606d60d4b0da953cab6e446dcfc6ccaa3b: Status 404 returned error can't find the container with id ff4154c6dbf042fe0f88e6aa873bc0606d60d4b0da953cab6e446dcfc6ccaa3b
Jul 24 18:19:58.856896 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.856860    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k" event={"ID":"62d82e8d-8ef6-4f4e-8f30-98973140f10c","Type":"ContainerStarted","Data":"132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9"}
Jul 24 18:19:58.857298 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.856913    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k" event={"ID":"62d82e8d-8ef6-4f4e-8f30-98973140f10c","Type":"ContainerStarted","Data":"ff4154c6dbf042fe0f88e6aa873bc0606d60d4b0da953cab6e446dcfc6ccaa3b"}
Jul 24 18:19:58.857298 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.856957    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:19:58.858447 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.858421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7" event={"ID":"c8424797-6952-4900-9833-903463a900e4","Type":"ContainerStarted","Data":"cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2"}
Jul 24 18:19:58.858579 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.858561    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:19:58.876176 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.876132    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k" podStartSLOduration=1.8761198430000001 podStartE2EDuration="1.876119843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:58.874530325 +0000 UTC m=+2389.991457680" watchObservedRunningTime="2026-07-24 18:19:58.876119843 +0000 UTC m=+2389.993047182"
Jul 24 18:19:58.897489 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:19:58.897435    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7" podStartSLOduration=2.440504694 podStartE2EDuration="2.897418012s" totalImagesPullingTime="456.913318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:19:58.895193327 +0000 UTC m=+2390.012120664" watchObservedRunningTime="2026-07-24 18:19:58.897418012 +0000 UTC m=+2390.014345345"
Jul 24 18:20:01.820870 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:01.820841    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:20:02.825220 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:02.825192    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:20:04.869709 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:04.869682    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:20:07.250162 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:07.250127    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:20:09.417850 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:09.417826    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:20:09.425511 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:09.425490    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:20:09.866052 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:09.866024    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:20:24.703900 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.703864    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"]
Jul 24 18:20:24.704474 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.704050    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg" podUID="dbbaefc9-56ec-4812-a395-f3e169710a12" containerName="maas-api" containerID="cri-o://5bda53ff80d709604e7a830f55fe2dd74bb675fdc0dd0d375454967048b10a79" gracePeriod=30
Jul 24 18:20:24.795366 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.795327    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"]
Jul 24 18:20:24.795641 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.795593    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg" podUID="c0f1fe51-94b6-4894-9c4b-4798a7693f82" containerName="payload-processing" containerID="cri-o://93660daada3b26426d78e1e1c5d894e0f1fc0a2b21f46efc5dba224b4a784699" gracePeriod=30
Jul 24 18:20:24.803859 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.803834    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"]
Jul 24 18:20:24.804059 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.804023    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8" podUID="00a84d83-2bce-4440-afa9-696169897e40" containerName="payload-pre-processing" containerID="cri-o://e3ce524ffecda8ce78acf4419998f0f83798fbf42f52dbe8752aec1e9cf58be6" gracePeriod=30
Jul 24 18:20:24.993503 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.993456    2551 generic.go:356] "Generic (PLEG): container finished" podID="c0f1fe51-94b6-4894-9c4b-4798a7693f82" containerID="93660daada3b26426d78e1e1c5d894e0f1fc0a2b21f46efc5dba224b4a784699" exitCode=0
Jul 24 18:20:24.993503 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.993492    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg" event={"ID":"c0f1fe51-94b6-4894-9c4b-4798a7693f82","Type":"ContainerDied","Data":"93660daada3b26426d78e1e1c5d894e0f1fc0a2b21f46efc5dba224b4a784699"}
Jul 24 18:20:24.995400 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.995352    2551 generic.go:356] "Generic (PLEG): container finished" podID="dbbaefc9-56ec-4812-a395-f3e169710a12" containerID="5bda53ff80d709604e7a830f55fe2dd74bb675fdc0dd0d375454967048b10a79" exitCode=0
Jul 24 18:20:24.995498 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.995407    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg" event={"ID":"dbbaefc9-56ec-4812-a395-f3e169710a12","Type":"ContainerDied","Data":"5bda53ff80d709604e7a830f55fe2dd74bb675fdc0dd0d375454967048b10a79"}
Jul 24 18:20:24.997136 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.997046    2551 generic.go:356] "Generic (PLEG): container finished" podID="00a84d83-2bce-4440-afa9-696169897e40" containerID="e3ce524ffecda8ce78acf4419998f0f83798fbf42f52dbe8752aec1e9cf58be6" exitCode=0
Jul 24 18:20:24.997136 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:24.997090    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8" event={"ID":"00a84d83-2bce-4440-afa9-696169897e40","Type":"ContainerDied","Data":"e3ce524ffecda8ce78acf4419998f0f83798fbf42f52dbe8752aec1e9cf58be6"}
Jul 24 18:20:25.057442 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.057413    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:20:25.076178 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.076150    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:20:25.090210 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.090183    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:20:25.167211 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167179    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbaefc9-56ec-4812-a395-f3e169710a12-openshift-service-ca\") pod \"dbbaefc9-56ec-4812-a395-f3e169710a12\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") "
Jul 24 18:20:25.167434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167224    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbbaefc9-56ec-4812-a395-f3e169710a12-maas-api-tls\") pod \"dbbaefc9-56ec-4812-a395-f3e169710a12\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") "
Jul 24 18:20:25.167434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167248    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f1fe51-94b6-4894-9c4b-4798a7693f82-plugins-config-volume\") pod \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") "
Jul 24 18:20:25.167434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167272    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5qln\" (UniqueName: \"kubernetes.io/projected/dbbaefc9-56ec-4812-a395-f3e169710a12-kube-api-access-q5qln\") pod \"dbbaefc9-56ec-4812-a395-f3e169710a12\" (UID: \"dbbaefc9-56ec-4812-a395-f3e169710a12\") "
Jul 24 18:20:25.167434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167394    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfns4\" (UniqueName: \"kubernetes.io/projected/c0f1fe51-94b6-4894-9c4b-4798a7693f82-kube-api-access-xfns4\") pod \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\" (UID: \"c0f1fe51-94b6-4894-9c4b-4798a7693f82\") "
Jul 24 18:20:25.167679 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167465    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a84d83-2bce-4440-afa9-696169897e40-plugins-config-volume\") pod \"00a84d83-2bce-4440-afa9-696169897e40\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") "
Jul 24 18:20:25.167679 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167494    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf7mj\" (UniqueName: \"kubernetes.io/projected/00a84d83-2bce-4440-afa9-696169897e40-kube-api-access-bf7mj\") pod \"00a84d83-2bce-4440-afa9-696169897e40\" (UID: \"00a84d83-2bce-4440-afa9-696169897e40\") "
Jul 24 18:20:25.167679 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167665    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbaefc9-56ec-4812-a395-f3e169710a12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbbaefc9-56ec-4812-a395-f3e169710a12" (UID: "dbbaefc9-56ec-4812-a395-f3e169710a12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:20:25.167826 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167681    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f1fe51-94b6-4894-9c4b-4798a7693f82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0f1fe51-94b6-4894-9c4b-4798a7693f82" (UID: "c0f1fe51-94b6-4894-9c4b-4798a7693f82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:20:25.167907 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.167880    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a84d83-2bce-4440-afa9-696169897e40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00a84d83-2bce-4440-afa9-696169897e40" (UID: "00a84d83-2bce-4440-afa9-696169897e40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:20:25.168109 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.168008    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbbaefc9-56ec-4812-a395-f3e169710a12-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:25.168204 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.168181    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f1fe51-94b6-4894-9c4b-4798a7693f82-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:25.168248 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.168209    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a84d83-2bce-4440-afa9-696169897e40-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:25.169644 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.169619    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f1fe51-94b6-4894-9c4b-4798a7693f82-kube-api-access-xfns4" (OuterVolumeSpecName: "kube-api-access-xfns4") pod "c0f1fe51-94b6-4894-9c4b-4798a7693f82" (UID: "c0f1fe51-94b6-4894-9c4b-4798a7693f82"). InnerVolumeSpecName "kube-api-access-xfns4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:20:25.169789 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.169705    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a84d83-2bce-4440-afa9-696169897e40-kube-api-access-bf7mj" (OuterVolumeSpecName: "kube-api-access-bf7mj") pod "00a84d83-2bce-4440-afa9-696169897e40" (UID: "00a84d83-2bce-4440-afa9-696169897e40"). InnerVolumeSpecName "kube-api-access-bf7mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:20:25.170059 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.170034    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbbaefc9-56ec-4812-a395-f3e169710a12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbbaefc9-56ec-4812-a395-f3e169710a12" (UID: "dbbaefc9-56ec-4812-a395-f3e169710a12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:20:25.170130 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.170036    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbaefc9-56ec-4812-a395-f3e169710a12-kube-api-access-q5qln" (OuterVolumeSpecName: "kube-api-access-q5qln") pod "dbbaefc9-56ec-4812-a395-f3e169710a12" (UID: "dbbaefc9-56ec-4812-a395-f3e169710a12"). InnerVolumeSpecName "kube-api-access-q5qln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:20:25.269282 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.269205    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbbaefc9-56ec-4812-a395-f3e169710a12-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:25.269282 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.269234    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qln\" (UniqueName: \"kubernetes.io/projected/dbbaefc9-56ec-4812-a395-f3e169710a12-kube-api-access-q5qln\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:25.269282 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.269244    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfns4\" (UniqueName: \"kubernetes.io/projected/c0f1fe51-94b6-4894-9c4b-4798a7693f82-kube-api-access-xfns4\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:25.269282 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:25.269252    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf7mj\" (UniqueName: \"kubernetes.io/projected/00a84d83-2bce-4440-afa9-696169897e40-kube-api-access-bf7mj\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:26.003568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.003477    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"
Jul 24 18:20:26.003568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.003484    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg" event={"ID":"dbbaefc9-56ec-4812-a395-f3e169710a12","Type":"ContainerDied","Data":"5f2b684ef96528d6251a497a36ea4d486c0e8ac5be371135878dc1525dcee6ea"}
Jul 24 18:20:26.003568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.003553    2551 scope.go:117] "RemoveContainer" containerID="5bda53ff80d709604e7a830f55fe2dd74bb675fdc0dd0d375454967048b10a79"
Jul 24 18:20:26.005063 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.005043    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"
Jul 24 18:20:26.005242 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.005061    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8" event={"ID":"00a84d83-2bce-4440-afa9-696169897e40","Type":"ContainerDied","Data":"0f13991dd600a193ede29acf235aaae38498d09e3b124822b410ed7eee83071c"}
Jul 24 18:20:26.006663 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.006641    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg" event={"ID":"c0f1fe51-94b6-4894-9c4b-4798a7693f82","Type":"ContainerDied","Data":"d0836306e231c715b8dbc95cc0fbde4e54fbf1c0ac0b982a7d811610a77e004f"}
Jul 24 18:20:26.006766 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.006674    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"
Jul 24 18:20:26.013728 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.013708    2551 scope.go:117] "RemoveContainer" containerID="e3ce524ffecda8ce78acf4419998f0f83798fbf42f52dbe8752aec1e9cf58be6"
Jul 24 18:20:26.021810 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.021791    2551 scope.go:117] "RemoveContainer" containerID="93660daada3b26426d78e1e1c5d894e0f1fc0a2b21f46efc5dba224b4a784699"
Jul 24 18:20:26.027493 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.027474    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"]
Jul 24 18:20:26.033461 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.033441    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-013528-7dcbcdc886-m8dlg"]
Jul 24 18:20:26.043784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.043763    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"]
Jul 24 18:20:26.047543 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.047523    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-013528-5c4858bbdb-lpzlg"]
Jul 24 18:20:26.057901 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.057882    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"]
Jul 24 18:20:26.063094 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:26.063075    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-013528-649fdd9496-dvmp8"]
Jul 24 18:20:27.299930 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:27.299899    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a84d83-2bce-4440-afa9-696169897e40" path="/var/lib/kubelet/pods/00a84d83-2bce-4440-afa9-696169897e40/volumes"
Jul 24 18:20:27.300314 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:27.300298    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f1fe51-94b6-4894-9c4b-4798a7693f82" path="/var/lib/kubelet/pods/c0f1fe51-94b6-4894-9c4b-4798a7693f82/volumes"
Jul 24 18:20:27.300706 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:27.300692    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbbaefc9-56ec-4812-a395-f3e169710a12" path="/var/lib/kubelet/pods/dbbaefc9-56ec-4812-a395-f3e169710a12/volumes"
Jul 24 18:20:27.540274 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:27.540238    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:20:51.698801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:51.698718    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"]
Jul 24 18:20:51.699252 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:51.698968    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7" podUID="c8424797-6952-4900-9833-903463a900e4" containerName="maas-api" containerID="cri-o://cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2" gracePeriod=30
Jul 24 18:20:51.758250 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:51.758217    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"]
Jul 24 18:20:51.758508 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:51.758482    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k" podUID="62d82e8d-8ef6-4f4e-8f30-98973140f10c" containerName="payload-processing" containerID="cri-o://132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9" gracePeriod=30
Jul 24 18:20:52.013072 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.013045    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:20:52.027616 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.027590    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tgzk\" (UniqueName: \"kubernetes.io/projected/c8424797-6952-4900-9833-903463a900e4-kube-api-access-9tgzk\") pod \"c8424797-6952-4900-9833-903463a900e4\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") "
Jul 24 18:20:52.027724 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.027661    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls\") pod \"c8424797-6952-4900-9833-903463a900e4\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") "
Jul 24 18:20:52.027724 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.027697    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8424797-6952-4900-9833-903463a900e4-openshift-service-ca\") pod \"c8424797-6952-4900-9833-903463a900e4\" (UID: \"c8424797-6952-4900-9833-903463a900e4\") "
Jul 24 18:20:52.028080 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.028052    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8424797-6952-4900-9833-903463a900e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8424797-6952-4900-9833-903463a900e4" (UID: "c8424797-6952-4900-9833-903463a900e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:20:52.029730 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.029687    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8424797-6952-4900-9833-903463a900e4-kube-api-access-9tgzk" (OuterVolumeSpecName: "kube-api-access-9tgzk") pod "c8424797-6952-4900-9833-903463a900e4" (UID: "c8424797-6952-4900-9833-903463a900e4"). InnerVolumeSpecName "kube-api-access-9tgzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:20:52.029811 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.029783    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8424797-6952-4900-9833-903463a900e4" (UID: "c8424797-6952-4900-9833-903463a900e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:20:52.044075 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.044045    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:20:52.128662 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.128631    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d82e8d-8ef6-4f4e-8f30-98973140f10c-plugins-config-volume\") pod \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") "
Jul 24 18:20:52.128825 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.128673    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t65k\" (UniqueName: \"kubernetes.io/projected/62d82e8d-8ef6-4f4e-8f30-98973140f10c-kube-api-access-4t65k\") pod \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\" (UID: \"62d82e8d-8ef6-4f4e-8f30-98973140f10c\") "
Jul 24 18:20:52.128908 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.128897    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8424797-6952-4900-9833-903463a900e4-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:52.128954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.128911    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8424797-6952-4900-9833-903463a900e4-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:52.128954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.128920    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgzk\" (UniqueName: \"kubernetes.io/projected/c8424797-6952-4900-9833-903463a900e4-kube-api-access-9tgzk\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:52.129023 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.128974    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d82e8d-8ef6-4f4e-8f30-98973140f10c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62d82e8d-8ef6-4f4e-8f30-98973140f10c" (UID: "62d82e8d-8ef6-4f4e-8f30-98973140f10c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:20:52.130855 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.130832    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d82e8d-8ef6-4f4e-8f30-98973140f10c-kube-api-access-4t65k" (OuterVolumeSpecName: "kube-api-access-4t65k") pod "62d82e8d-8ef6-4f4e-8f30-98973140f10c" (UID: "62d82e8d-8ef6-4f4e-8f30-98973140f10c"). InnerVolumeSpecName "kube-api-access-4t65k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:20:52.144643 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.144614    2551 generic.go:356] "Generic (PLEG): container finished" podID="62d82e8d-8ef6-4f4e-8f30-98973140f10c" containerID="132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9" exitCode=0
Jul 24 18:20:52.144784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.144661    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"
Jul 24 18:20:52.144784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.144699    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k" event={"ID":"62d82e8d-8ef6-4f4e-8f30-98973140f10c","Type":"ContainerDied","Data":"132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9"}
Jul 24 18:20:52.144784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.144757    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k" event={"ID":"62d82e8d-8ef6-4f4e-8f30-98973140f10c","Type":"ContainerDied","Data":"ff4154c6dbf042fe0f88e6aa873bc0606d60d4b0da953cab6e446dcfc6ccaa3b"}
Jul 24 18:20:52.144944 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.144791    2551 scope.go:117] "RemoveContainer" containerID="132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9"
Jul 24 18:20:52.146070 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.146044    2551 generic.go:356] "Generic (PLEG): container finished" podID="c8424797-6952-4900-9833-903463a900e4" containerID="cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2" exitCode=0
Jul 24 18:20:52.146185 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.146136    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"
Jul 24 18:20:52.146256 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.146136    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7" event={"ID":"c8424797-6952-4900-9833-903463a900e4","Type":"ContainerDied","Data":"cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2"}
Jul 24 18:20:52.146538 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.146510    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7" event={"ID":"c8424797-6952-4900-9833-903463a900e4","Type":"ContainerDied","Data":"14c04f1f5b3ea0d4e884c5760d53e1b62c58480fcd676594116318004e743958"}
Jul 24 18:20:52.162280 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.162262    2551 scope.go:117] "RemoveContainer" containerID="132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9"
Jul 24 18:20:52.162595 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:20:52.162576    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9\": container with ID starting with 132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9 not found: ID does not exist" containerID="132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9"
Jul 24 18:20:52.162668 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.162600    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9"} err="failed to get container status \"132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9\": rpc error: code = NotFound desc = could not find container \"132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9\": container with ID starting with 132cb1378fcb90ed857e831a950c9c10d2781f988f65bbf175bfc5b12e5dffa9 not found: ID does not exist"
Jul 24 18:20:52.162668 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.162615    2551 scope.go:117] "RemoveContainer" containerID="cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2"
Jul 24 18:20:52.171826 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.171621    2551 scope.go:117] "RemoveContainer" containerID="cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2"
Jul 24 18:20:52.172028 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:20:52.172008    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2\": container with ID starting with cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2 not found: ID does not exist" containerID="cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2"
Jul 24 18:20:52.172089 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.172036    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2"} err="failed to get container status \"cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2\": rpc error: code = NotFound desc = could not find container \"cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2\": container with ID starting with cd8f2f04c66bd31a18004a1ff5abc69e59144bebe7be822a88ad0cc6ad9040e2 not found: ID does not exist"
Jul 24 18:20:52.173857 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.173838    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"]
Jul 24 18:20:52.181218 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.181187    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1a9df-f79d6c8bd-rw95k"]
Jul 24 18:20:52.191690 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.191653    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"]
Jul 24 18:20:52.193173 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.193153    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d1a9df-6698f9fc79-6qfj7"]
Jul 24 18:20:52.229419 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.229324    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d82e8d-8ef6-4f4e-8f30-98973140f10c-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:52.229419 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:52.229351    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t65k\" (UniqueName: \"kubernetes.io/projected/62d82e8d-8ef6-4f4e-8f30-98973140f10c-kube-api-access-4t65k\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:20:53.299985 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:53.299955    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d82e8d-8ef6-4f4e-8f30-98973140f10c" path="/var/lib/kubelet/pods/62d82e8d-8ef6-4f4e-8f30-98973140f10c/volumes"
Jul 24 18:20:53.300359 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:53.300331    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8424797-6952-4900-9833-903463a900e4" path="/var/lib/kubelet/pods/c8424797-6952-4900-9833-903463a900e4/volumes"
Jul 24 18:20:56.453994 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:20:56.453951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:21:07.039160 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039121    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"]
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039750    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8424797-6952-4900-9833-903463a900e4" containerName="maas-api"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039766    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8424797-6952-4900-9833-903463a900e4" containerName="maas-api"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039782    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f1fe51-94b6-4894-9c4b-4798a7693f82" containerName="payload-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039787    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f1fe51-94b6-4894-9c4b-4798a7693f82" containerName="payload-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039795    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbbaefc9-56ec-4812-a395-f3e169710a12" containerName="maas-api"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039802    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbbaefc9-56ec-4812-a395-f3e169710a12" containerName="maas-api"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039809    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d82e8d-8ef6-4f4e-8f30-98973140f10c" containerName="payload-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039814    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d82e8d-8ef6-4f4e-8f30-98973140f10c" containerName="payload-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039824    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a84d83-2bce-4440-afa9-696169897e40" containerName="payload-pre-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039829    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a84d83-2bce-4440-afa9-696169897e40" containerName="payload-pre-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039902    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d82e8d-8ef6-4f4e-8f30-98973140f10c" containerName="payload-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039910    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f1fe51-94b6-4894-9c4b-4798a7693f82" containerName="payload-processing"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039918    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbbaefc9-56ec-4812-a395-f3e169710a12" containerName="maas-api"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039923    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8424797-6952-4900-9833-903463a900e4" containerName="maas-api"
Jul 24 18:21:07.041598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.039929    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a84d83-2bce-4440-afa9-696169897e40" containerName="payload-pre-processing"
Jul 24 18:21:07.042835 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.042814    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.045691 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.045487    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 18:21:07.045953 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.045930    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 18:21:07.046086 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.046020    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ppc8g\""
Jul 24 18:21:07.046086 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.046020    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-b90928-serving-cert\""
Jul 24 18:21:07.054816 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.054794    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"]
Jul 24 18:21:07.166245 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.166212    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44p4j\" (UniqueName: \"kubernetes.io/projected/5b85d54a-0332-40ff-9c45-3dec141c2258-kube-api-access-44p4j\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.166475 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.166263    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.166475 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.166447    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85d54a-0332-40ff-9c45-3dec141c2258-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.267856 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.267823    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85d54a-0332-40ff-9c45-3dec141c2258-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.268014 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.267866    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44p4j\" (UniqueName: \"kubernetes.io/projected/5b85d54a-0332-40ff-9c45-3dec141c2258-kube-api-access-44p4j\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.268014 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.267980    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.268168 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:21:07.268145    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-serving-cert: secret "maas-api-e2e-shared-a-b90928-serving-cert" not found
Jul 24 18:21:07.268242 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:21:07.268229    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls podName:5b85d54a-0332-40ff-9c45-3dec141c2258 nodeName:}" failed. No retries permitted until 2026-07-24 18:21:07.768206449 +0000 UTC m=+2458.885133782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls") pod "maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b" (UID: "5b85d54a-0332-40ff-9c45-3dec141c2258") : secret "maas-api-e2e-shared-a-b90928-serving-cert" not found
Jul 24 18:21:07.268549 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.268525    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85d54a-0332-40ff-9c45-3dec141c2258-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.283829 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.283802    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44p4j\" (UniqueName: \"kubernetes.io/projected/5b85d54a-0332-40ff-9c45-3dec141c2258-kube-api-access-44p4j\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.649318 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.649282    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"]
Jul 24 18:21:07.652965 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.652947    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:07.655893 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.655876    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-b90928\""
Jul 24 18:21:07.666952 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.666928    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"]
Jul 24 18:21:07.692431 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.692394    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"]
Jul 24 18:21:07.695726 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.695704    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:07.698146 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.698128    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-b90928-dockercfg-4t74k\""
Jul 24 18:21:07.705613 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.705591    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"]
Jul 24 18:21:07.773139 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.773101    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpcq\" (UniqueName: \"kubernetes.io/projected/1544e5d0-2408-49bb-b583-971407d62377-kube-api-access-rxpcq\") pod \"payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:07.773309 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.773157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxkr\" (UniqueName: \"kubernetes.io/projected/01051c77-e1e5-4938-8283-8746530c9971-kube-api-access-ldxkr\") pod \"payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:07.773309 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.773236    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.773417 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.773324    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01051c77-e1e5-4938-8283-8746530c9971-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:07.773417 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.773404    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1544e5d0-2408-49bb-b583-971407d62377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:07.776077 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.776057    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls\") pod \"maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.874082 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.874039    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1544e5d0-2408-49bb-b583-971407d62377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:07.874253 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.874102    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpcq\" (UniqueName: \"kubernetes.io/projected/1544e5d0-2408-49bb-b583-971407d62377-kube-api-access-rxpcq\") pod \"payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:07.874253 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.874126    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxkr\" (UniqueName: \"kubernetes.io/projected/01051c77-e1e5-4938-8283-8746530c9971-kube-api-access-ldxkr\") pod \"payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:07.874253 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.874187    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01051c77-e1e5-4938-8283-8746530c9971-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:07.874832 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.874810    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1544e5d0-2408-49bb-b583-971407d62377-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:07.874883 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.874866    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01051c77-e1e5-4938-8283-8746530c9971-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:07.882950 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.882919    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxkr\" (UniqueName: \"kubernetes.io/projected/01051c77-e1e5-4938-8283-8746530c9971-kube-api-access-ldxkr\") pod \"payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") " pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:07.882950 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.882944    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpcq\" (UniqueName: \"kubernetes.io/projected/1544e5d0-2408-49bb-b583-971407d62377-kube-api-access-rxpcq\") pod \"payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:07.954938 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.954843    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:07.963122 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:07.963093    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:08.007300 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.007250    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:08.120644 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.120603    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"]
Jul 24 18:21:08.124146 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:21:08.124089    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b85d54a_0332_40ff_9c45_3dec141c2258.slice/crio-1fc263f8bd97d41d0c814b90d383a3bbeaeec983c2a8a72146f7d3a9c332e35b WatchSource:0}: Error finding container 1fc263f8bd97d41d0c814b90d383a3bbeaeec983c2a8a72146f7d3a9c332e35b: Status 404 returned error can't find the container with id 1fc263f8bd97d41d0c814b90d383a3bbeaeec983c2a8a72146f7d3a9c332e35b
Jul 24 18:21:08.150229 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.150113    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"]
Jul 24 18:21:08.152952 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:21:08.152922    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1544e5d0_2408_49bb_b583_971407d62377.slice/crio-7fcc843d3dc61a67aa65f723a483b4427539e58f65e3c7d8370df757566839bd WatchSource:0}: Error finding container 7fcc843d3dc61a67aa65f723a483b4427539e58f65e3c7d8370df757566839bd: Status 404 returned error can't find the container with id 7fcc843d3dc61a67aa65f723a483b4427539e58f65e3c7d8370df757566839bd
Jul 24 18:21:08.176624 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.176595    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"]
Jul 24 18:21:08.184530 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:21:08.184503    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01051c77_e1e5_4938_8283_8746530c9971.slice/crio-ad4d93afcbea6d37583d4eda9ab1155293a81c0c72ac07203156e38671ec86a5 WatchSource:0}: Error finding container ad4d93afcbea6d37583d4eda9ab1155293a81c0c72ac07203156e38671ec86a5: Status 404 returned error can't find the container with id ad4d93afcbea6d37583d4eda9ab1155293a81c0c72ac07203156e38671ec86a5
Jul 24 18:21:08.229168 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.229139    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x" event={"ID":"1544e5d0-2408-49bb-b583-971407d62377","Type":"ContainerStarted","Data":"90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419"}
Jul 24 18:21:08.229284 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.229196    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x" event={"ID":"1544e5d0-2408-49bb-b583-971407d62377","Type":"ContainerStarted","Data":"7fcc843d3dc61a67aa65f723a483b4427539e58f65e3c7d8370df757566839bd"}
Jul 24 18:21:08.229284 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.229226    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:08.230613 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.230590    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b" event={"ID":"5b85d54a-0332-40ff-9c45-3dec141c2258","Type":"ContainerStarted","Data":"1fc263f8bd97d41d0c814b90d383a3bbeaeec983c2a8a72146f7d3a9c332e35b"}
Jul 24 18:21:08.232264 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.232240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd" event={"ID":"01051c77-e1e5-4938-8283-8746530c9971","Type":"ContainerStarted","Data":"ad4d93afcbea6d37583d4eda9ab1155293a81c0c72ac07203156e38671ec86a5"}
Jul 24 18:21:08.247287 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:08.247244    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x" podStartSLOduration=1.24723017 podStartE2EDuration="1.24723017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:21:08.245057964 +0000 UTC m=+2459.361985304" watchObservedRunningTime="2026-07-24 18:21:08.24723017 +0000 UTC m=+2459.364157509"
Jul 24 18:21:09.247914 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:09.247867    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b" event={"ID":"5b85d54a-0332-40ff-9c45-3dec141c2258","Type":"ContainerStarted","Data":"c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb"}
Jul 24 18:21:09.248448 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:09.247972    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:09.249620 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:09.249596    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd" event={"ID":"01051c77-e1e5-4938-8283-8746530c9971","Type":"ContainerStarted","Data":"339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f"}
Jul 24 18:21:09.249789 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:09.249773    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:09.266784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:09.266737    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b" podStartSLOduration=1.7510810970000001 podStartE2EDuration="2.266722373s" totalImagesPullingTime="515.641276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:21:09.265307857 +0000 UTC m=+2460.382235196" watchObservedRunningTime="2026-07-24 18:21:09.266722373 +0000 UTC m=+2460.383649713"
Jul 24 18:21:09.281909 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:09.281867    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd" podStartSLOduration=2.281852819 podStartE2EDuration="2.281852819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:21:09.281123543 +0000 UTC m=+2460.398050899" watchObservedRunningTime="2026-07-24 18:21:09.281852819 +0000 UTC m=+2460.398780160"
Jul 24 18:21:15.260448 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.260414    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:21:15.615794 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.615759    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"]
Jul 24 18:21:15.619419 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.619400    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.622785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.622765    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-88d4a7-serving-cert\""
Jul 24 18:21:15.639148 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.639123    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"]
Jul 24 18:21:15.735648 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.735616    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:21:15.749528 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.749498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr92v\" (UniqueName: \"kubernetes.io/projected/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-kube-api-access-jr92v\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.749706 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.749557    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.749706 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.749592    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.850427 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.850366    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.850601 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.850461    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.850601 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.850533    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr92v\" (UniqueName: \"kubernetes.io/projected/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-kube-api-access-jr92v\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.850675 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:21:15.850624    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-serving-cert: secret "maas-api-e2e-shared-b-88d4a7-serving-cert" not found
Jul 24 18:21:15.850744 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:21:15.850719    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls podName:f0547ac1-2ebf-4181-9a5d-29d1062fdfb6 nodeName:}" failed. No retries permitted until 2026-07-24 18:21:16.350695474 +0000 UTC m=+2467.467622796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls") pod "maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4" (UID: "f0547ac1-2ebf-4181-9a5d-29d1062fdfb6") : secret "maas-api-e2e-shared-b-88d4a7-serving-cert" not found
Jul 24 18:21:15.851142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.851118    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:15.860011 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:15.859987    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr92v\" (UniqueName: \"kubernetes.io/projected/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-kube-api-access-jr92v\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:16.352436 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.352403    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"]
Jul 24 18:21:16.356143 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.356125    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.356479 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.356454    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:16.358972 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.358950    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-88d4a7\""
Jul 24 18:21:16.359072 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.358976    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls\") pod \"maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:16.368364 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.368341    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"]
Jul 24 18:21:16.457337 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.457296    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.457531 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.457428    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b4qk\" (UniqueName: \"kubernetes.io/projected/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-kube-api-access-2b4qk\") pod \"payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.516171 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.516135    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"]
Jul 24 18:21:16.519933 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.519915    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.522944 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.522915    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-88d4a7-dockercfg-pw478\""
Jul 24 18:21:16.529249 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.529158    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:16.531277 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.531250    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"]
Jul 24 18:21:16.558019 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.557986    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4qk\" (UniqueName: \"kubernetes.io/projected/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-kube-api-access-2b4qk\") pod \"payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.558166 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.558049    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.558808 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.558788    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.567210 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.567182    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4qk\" (UniqueName: \"kubernetes.io/projected/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-kube-api-access-2b4qk\") pod \"payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.659294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.659261    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmtn\" (UniqueName: \"kubernetes.io/projected/24da083a-e97e-4dea-ab12-a55f08463948-kube-api-access-gtmtn\") pod \"payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") " pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.659294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.659308    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24da083a-e97e-4dea-ab12-a55f08463948-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") " pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.676083 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.676037    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:16.679180 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.679153    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"]
Jul 24 18:21:16.680610 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:21:16.680582    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0547ac1_2ebf_4181_9a5d_29d1062fdfb6.slice/crio-de0b19410c3610ebffaba9e8218974f60dc506e2a5e779418a238a34b754c490 WatchSource:0}: Error finding container de0b19410c3610ebffaba9e8218974f60dc506e2a5e779418a238a34b754c490: Status 404 returned error can't find the container with id de0b19410c3610ebffaba9e8218974f60dc506e2a5e779418a238a34b754c490
Jul 24 18:21:16.760900 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.760861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmtn\" (UniqueName: \"kubernetes.io/projected/24da083a-e97e-4dea-ab12-a55f08463948-kube-api-access-gtmtn\") pod \"payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") " pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.761079 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.760909    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24da083a-e97e-4dea-ab12-a55f08463948-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") " pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.761626 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.761603    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24da083a-e97e-4dea-ab12-a55f08463948-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") " pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.770293 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.770264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmtn\" (UniqueName: \"kubernetes.io/projected/24da083a-e97e-4dea-ab12-a55f08463948-kube-api-access-gtmtn\") pod \"payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") " pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.829166 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.829140    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"]
Jul 24 18:21:16.830849 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.830822    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:16.831443 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:21:16.831403    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06bb3fc_d5af_41f0_9ebb_f71c6381f672.slice/crio-825afba92b835e60a9bc0624ca538f403d474677f314a0b47c014d0fc137ae95 WatchSource:0}: Error finding container 825afba92b835e60a9bc0624ca538f403d474677f314a0b47c014d0fc137ae95: Status 404 returned error can't find the container with id 825afba92b835e60a9bc0624ca538f403d474677f314a0b47c014d0fc137ae95
Jul 24 18:21:16.985677 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:16.985648    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"]
Jul 24 18:21:16.986263 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:21:16.986236    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24da083a_e97e_4dea_ab12_a55f08463948.slice/crio-286f17b559e0ce0c4ed4006d6176d70f4483c69c6acfb6ca593f9dfea37e49a2 WatchSource:0}: Error finding container 286f17b559e0ce0c4ed4006d6176d70f4483c69c6acfb6ca593f9dfea37e49a2: Status 404 returned error can't find the container with id 286f17b559e0ce0c4ed4006d6176d70f4483c69c6acfb6ca593f9dfea37e49a2
Jul 24 18:21:17.302615 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302577    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm" event={"ID":"f06bb3fc-d5af-41f0-9ebb-f71c6381f672","Type":"ContainerStarted","Data":"ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f"}
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302634    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302655    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302677    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm" event={"ID":"f06bb3fc-d5af-41f0-9ebb-f71c6381f672","Type":"ContainerStarted","Data":"825afba92b835e60a9bc0624ca538f403d474677f314a0b47c014d0fc137ae95"}
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302701    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4" event={"ID":"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6","Type":"ContainerStarted","Data":"f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63"}
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4" event={"ID":"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6","Type":"ContainerStarted","Data":"de0b19410c3610ebffaba9e8218974f60dc506e2a5e779418a238a34b754c490"}
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302731    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302748    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q" event={"ID":"24da083a-e97e-4dea-ab12-a55f08463948","Type":"ContainerStarted","Data":"06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09"}
Jul 24 18:21:17.302802 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.302771    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q" event={"ID":"24da083a-e97e-4dea-ab12-a55f08463948","Type":"ContainerStarted","Data":"286f17b559e0ce0c4ed4006d6176d70f4483c69c6acfb6ca593f9dfea37e49a2"}
Jul 24 18:21:17.316502 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.316450    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm" podStartSLOduration=1.3164338500000001 podStartE2EDuration="1.31643385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:21:17.314569995 +0000 UTC m=+2468.431497360" watchObservedRunningTime="2026-07-24 18:21:17.31643385 +0000 UTC m=+2468.433361193"
Jul 24 18:21:17.332470 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.332417    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4" podStartSLOduration=2.035211754 podStartE2EDuration="2.332370013s" totalImagesPullingTime="297.158259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:21:17.330973306 +0000 UTC m=+2468.447900659" watchObservedRunningTime="2026-07-24 18:21:17.332370013 +0000 UTC m=+2468.449297352"
Jul 24 18:21:17.353482 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:17.353433    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q" podStartSLOduration=1.353418105 podStartE2EDuration="1.353418105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:21:17.351297222 +0000 UTC m=+2468.468224562" watchObservedRunningTime="2026-07-24 18:21:17.353418105 +0000 UTC m=+2468.470345445"
Jul 24 18:21:19.251446 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:19.251398    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x" podUID="1544e5d0-2408-49bb-b583-971407d62377" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.66:9004: connect: connection refused"
Jul 24 18:21:20.257476 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:20.257439    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:21:24.314978 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:24.314947    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:21:27.560070 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:27.560026    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:21:28.307274 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:28.307245    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:21:28.307562 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:28.307469    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:21:29.250528 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:29.250496    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:21:40.110680 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.110643    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"]
Jul 24 18:21:40.115809 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.115772    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.118021 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.117993    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b90928\"/\"kube-root-ca.crt\""
Jul 24 18:21:40.118913 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.118888    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b90928\"/\"openshift-service-ca.crt\""
Jul 24 18:21:40.119096 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.119075    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b90928\"/\"auth-test-model-b90928-kserve-self-signed-certs\""
Jul 24 18:21:40.119274 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.119116    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b90928\"/\"default-dockercfg-ld54t\""
Jul 24 18:21:40.125179 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.124787    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"]
Jul 24 18:21:40.190831 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.190794    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-dshm\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.190831 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.190840    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rqf\" (UniqueName: \"kubernetes.io/projected/a379665a-f9f8-4f26-9e7a-c57099db98e2-kube-api-access-r4rqf\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.191071 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.190931    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-home\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.191071 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.191006    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-model-cache\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.191071 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.191027    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-tmp-dir\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.191071 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.191045    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a379665a-f9f8-4f26-9e7a-c57099db98e2-tls-certs\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.291668 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.291632    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-model-cache\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.291668 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.291674    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-tmp-dir\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.291949 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.291693    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a379665a-f9f8-4f26-9e7a-c57099db98e2-tls-certs\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.291949 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.291712    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-dshm\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.291949 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.291749    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rqf\" (UniqueName: \"kubernetes.io/projected/a379665a-f9f8-4f26-9e7a-c57099db98e2-kube-api-access-r4rqf\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.291949 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.291817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-home\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.292169 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.292093    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-tmp-dir\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.292398 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.292351    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-home\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.292462 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.292446    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-model-cache\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.294638 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.294605    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-dshm\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.294996 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.294978    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a379665a-f9f8-4f26-9e7a-c57099db98e2-tls-certs\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.300783 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.300749    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rqf\" (UniqueName: \"kubernetes.io/projected/a379665a-f9f8-4f26-9e7a-c57099db98e2-kube-api-access-r4rqf\") pod \"auth-test-model-b90928-kserve-959c44699-dtbk2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") " pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.427787 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.427686    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:40.577358 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:40.577336    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"]
Jul 24 18:21:41.422483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:41.422436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2" event={"ID":"a379665a-f9f8-4f26-9e7a-c57099db98e2","Type":"ContainerStarted","Data":"4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515"}
Jul 24 18:21:41.422483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:41.422487    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2" event={"ID":"a379665a-f9f8-4f26-9e7a-c57099db98e2","Type":"ContainerStarted","Data":"f4e49e3fe73f40e65d47d009425f19080b20b227f2a3ae9d0b3ffd2665d145b3"}
Jul 24 18:21:41.443137 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:41.443080    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2" podStartSLOduration=1.443064093 podStartE2EDuration="1.443064093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:21:41.441262444 +0000 UTC m=+2492.558189784" watchObservedRunningTime="2026-07-24 18:21:41.443064093 +0000 UTC m=+2492.559991435"
Jul 24 18:21:43.650281 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:43.650237    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:21:50.428362 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:50.428326    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:50.428833 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:50.428400    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:50.446654 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:50.446620    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:21:50.479388 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:21:50.479338    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:22:03.417659 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.415690    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"]
Jul 24 18:22:03.429585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.429551    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"]
Jul 24 18:22:03.429762 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.429668    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.434047 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.434026    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"kube-root-ca.crt\""
Jul 24 18:22:03.434202 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.434141    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"auth-test-model-88d4a7-kserve-self-signed-certs\""
Jul 24 18:22:03.434924 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.434904    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"openshift-service-ca.crt\""
Jul 24 18:22:03.435268 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.435246    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"default-dockercfg-lvjgj\""
Jul 24 18:22:03.523114 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.523076    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-home\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.523290 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.523209    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-tmp-dir\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.523290 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.523250    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028d91c9-18c0-4f77-99f1-d46afee8282e-tls-certs\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.523290 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.523271    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-dshm\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.523437 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.523297    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-model-cache\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.523437 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.523355    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrlf\" (UniqueName: \"kubernetes.io/projected/028d91c9-18c0-4f77-99f1-d46afee8282e-kube-api-access-nfrlf\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.623905 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.623869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-tmp-dir\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624092 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.623914    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028d91c9-18c0-4f77-99f1-d46afee8282e-tls-certs\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624092 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.623933    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-dshm\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624092 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.623955    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-model-cache\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624092 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.623994    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrlf\" (UniqueName: \"kubernetes.io/projected/028d91c9-18c0-4f77-99f1-d46afee8282e-kube-api-access-nfrlf\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624092 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.624031    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-home\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.624401    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-tmp-dir\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624620 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.624467    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-model-cache\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.624723 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.624545    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-home\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.626540 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.626512    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-dshm\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.626686 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.626665    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028d91c9-18c0-4f77-99f1-d46afee8282e-tls-certs\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.632524 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.632500    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrlf\" (UniqueName: \"kubernetes.io/projected/028d91c9-18c0-4f77-99f1-d46afee8282e-kube-api-access-nfrlf\") pod \"auth-test-model-88d4a7-kserve-5595cc6687-mlqqx\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") " pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.740697 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.740610    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:03.885287 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:03.885253    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"]
Jul 24 18:22:03.888496 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:22:03.888451    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod028d91c9_18c0_4f77_99f1_d46afee8282e.slice/crio-26e5a592c15f537e63051e528110f237220001f8ff64b408c4d15b1522f348f2 WatchSource:0}: Error finding container 26e5a592c15f537e63051e528110f237220001f8ff64b408c4d15b1522f348f2: Status 404 returned error can't find the container with id 26e5a592c15f537e63051e528110f237220001f8ff64b408c4d15b1522f348f2
Jul 24 18:22:04.541905 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:04.541868    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx" event={"ID":"028d91c9-18c0-4f77-99f1-d46afee8282e","Type":"ContainerStarted","Data":"609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4"}
Jul 24 18:22:04.541905 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:04.541911    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx" event={"ID":"028d91c9-18c0-4f77-99f1-d46afee8282e","Type":"ContainerStarted","Data":"26e5a592c15f537e63051e528110f237220001f8ff64b408c4d15b1522f348f2"}
Jul 24 18:22:04.564131 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:04.564080    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx" podStartSLOduration=1.564061835 podStartE2EDuration="1.564061835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:22:04.562118476 +0000 UTC m=+2515.679045838" watchObservedRunningTime="2026-07-24 18:22:04.564061835 +0000 UTC m=+2515.680989203"
Jul 24 18:22:07.752887 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:07.752849    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:22:13.740911 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:13.740880    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:13.741307 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:13.740946    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:13.753368 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:13.753343    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:14.606050 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:14.606016    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:22:22.451616 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:22.451564    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:22:38.251028 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:38.250992    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:22:57.859844 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:22:57.859807    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:23:16.960277 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:23:16.960239    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:23:37.266192 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:23:37.266112    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:23:55.454964 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:23:55.454926    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:24:06.738897 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:06.738861    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"]
Jul 24 18:24:07.523047 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:07.523010    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"]
Jul 24 18:24:11.144039 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:11.143998    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:24:21.741625 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.741587    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"]
Jul 24 18:24:21.745423 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.745402    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.747580 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.747562    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b90928\"/\"sub-test-model-b90928-kserve-self-signed-certs\""
Jul 24 18:24:21.759236 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.759184    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2" podUID="a379665a-f9f8-4f26-9e7a-c57099db98e2" containerName="main" containerID="cri-o://4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515" gracePeriod=45
Jul 24 18:24:21.772155 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.772123    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"]
Jul 24 18:24:21.801973 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.801942    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxmb\" (UniqueName: \"kubernetes.io/projected/f65bf21c-a430-468a-a999-0773d5e41135-kube-api-access-jdxmb\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.802139 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.801991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-tmp-dir\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.802191 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.802127    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65bf21c-a430-468a-a999-0773d5e41135-tls-certs\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.802241 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.802201    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-dshm\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.802241 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.802225    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-home\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.802317 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.802250    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-model-cache\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.903744 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.903701    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxmb\" (UniqueName: \"kubernetes.io/projected/f65bf21c-a430-468a-a999-0773d5e41135-kube-api-access-jdxmb\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.903959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.903791    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-tmp-dir\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.903959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.903885    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65bf21c-a430-468a-a999-0773d5e41135-tls-certs\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.903959 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.903943    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-dshm\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.904158 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.903982    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-home\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.904158 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.904024    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-model-cache\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.904274 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.904251    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-tmp-dir\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.904335 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.904320    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-home\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.904556 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.904478    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-model-cache\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.906533 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.906507    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-dshm\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.906907 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.906882    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65bf21c-a430-468a-a999-0773d5e41135-tls-certs\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:21.912047 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:21.912021    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxmb\" (UniqueName: \"kubernetes.io/projected/f65bf21c-a430-468a-a999-0773d5e41135-kube-api-access-jdxmb\") pod \"sub-test-model-b90928-kserve-86d4b58fb7-rlpcr\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") " pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:22.024076 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.024046    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:24:22.054783 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.054740    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106349    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-model-cache\") pod \"a379665a-f9f8-4f26-9e7a-c57099db98e2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") "
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106431    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a379665a-f9f8-4f26-9e7a-c57099db98e2-tls-certs\") pod \"a379665a-f9f8-4f26-9e7a-c57099db98e2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") "
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106470    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-home\") pod \"a379665a-f9f8-4f26-9e7a-c57099db98e2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") "
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106501    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4rqf\" (UniqueName: \"kubernetes.io/projected/a379665a-f9f8-4f26-9e7a-c57099db98e2-kube-api-access-r4rqf\") pod \"a379665a-f9f8-4f26-9e7a-c57099db98e2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") "
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106566    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-tmp-dir\") pod \"a379665a-f9f8-4f26-9e7a-c57099db98e2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") "
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106623    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a379665a-f9f8-4f26-9e7a-c57099db98e2" (UID: "a379665a-f9f8-4f26-9e7a-c57099db98e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106693    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-dshm\") pod \"a379665a-f9f8-4f26-9e7a-c57099db98e2\" (UID: \"a379665a-f9f8-4f26-9e7a-c57099db98e2\") "
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.106795    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-home" (OuterVolumeSpecName: "home") pod "a379665a-f9f8-4f26-9e7a-c57099db98e2" (UID: "a379665a-f9f8-4f26-9e7a-c57099db98e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.107046    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.107098 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.107067    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.108301 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.107182    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a379665a-f9f8-4f26-9e7a-c57099db98e2" (UID: "a379665a-f9f8-4f26-9e7a-c57099db98e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:22.110093 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.110033    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a379665a-f9f8-4f26-9e7a-c57099db98e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a379665a-f9f8-4f26-9e7a-c57099db98e2" (UID: "a379665a-f9f8-4f26-9e7a-c57099db98e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:24:22.110198 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.110128    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a379665a-f9f8-4f26-9e7a-c57099db98e2-kube-api-access-r4rqf" (OuterVolumeSpecName: "kube-api-access-r4rqf") pod "a379665a-f9f8-4f26-9e7a-c57099db98e2" (UID: "a379665a-f9f8-4f26-9e7a-c57099db98e2"). InnerVolumeSpecName "kube-api-access-r4rqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:24:22.110693 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.110659    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-dshm" (OuterVolumeSpecName: "dshm") pod "a379665a-f9f8-4f26-9e7a-c57099db98e2" (UID: "a379665a-f9f8-4f26-9e7a-c57099db98e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:22.193670 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.193642    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"]
Jul 24 18:24:22.195800 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:24:22.195771    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf65bf21c_a430_468a_a999_0773d5e41135.slice/crio-ba45a9cdcaa4cad40c7968a719c4c5f016631d1bd212d68c1383c783c15f3546 WatchSource:0}: Error finding container ba45a9cdcaa4cad40c7968a719c4c5f016631d1bd212d68c1383c783c15f3546: Status 404 returned error can't find the container with id ba45a9cdcaa4cad40c7968a719c4c5f016631d1bd212d68c1383c783c15f3546
Jul 24 18:24:22.208422 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.208361    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.208535 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.208441    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a379665a-f9f8-4f26-9e7a-c57099db98e2-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.208535 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.208453    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4rqf\" (UniqueName: \"kubernetes.io/projected/a379665a-f9f8-4f26-9e7a-c57099db98e2-kube-api-access-r4rqf\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.208535 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.208462    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a379665a-f9f8-4f26-9e7a-c57099db98e2-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.252723 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.252650    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr" event={"ID":"f65bf21c-a430-468a-a999-0773d5e41135","Type":"ContainerStarted","Data":"ba45a9cdcaa4cad40c7968a719c4c5f016631d1bd212d68c1383c783c15f3546"}
Jul 24 18:24:22.253993 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.253969    2551 generic.go:356] "Generic (PLEG): container finished" podID="a379665a-f9f8-4f26-9e7a-c57099db98e2" containerID="4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515" exitCode=0
Jul 24 18:24:22.254151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.254077    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"
Jul 24 18:24:22.254363 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.254074    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2" event={"ID":"a379665a-f9f8-4f26-9e7a-c57099db98e2","Type":"ContainerDied","Data":"4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515"}
Jul 24 18:24:22.254512 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.254415    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2" event={"ID":"a379665a-f9f8-4f26-9e7a-c57099db98e2","Type":"ContainerDied","Data":"f4e49e3fe73f40e65d47d009425f19080b20b227f2a3ae9d0b3ffd2665d145b3"}
Jul 24 18:24:22.254512 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.254454    2551 scope.go:117] "RemoveContainer" containerID="4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515"
Jul 24 18:24:22.264832 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.264800    2551 scope.go:117] "RemoveContainer" containerID="4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515"
Jul 24 18:24:22.265099 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:24:22.265078    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515\": container with ID starting with 4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515 not found: ID does not exist" containerID="4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515"
Jul 24 18:24:22.265182 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.265112    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515"} err="failed to get container status \"4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515\": rpc error: code = NotFound desc = could not find container \"4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515\": container with ID starting with 4d764655a6f146b2c3ecf686bf6a0b99167f009715e5620552b41a332c750515 not found: ID does not exist"
Jul 24 18:24:22.277511 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.277459    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"]
Jul 24 18:24:22.280580 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.280556    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/auth-test-model-b90928-kserve-959c44699-dtbk2"]
Jul 24 18:24:22.543107 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.543008    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx" podUID="028d91c9-18c0-4f77-99f1-d46afee8282e" containerName="main" containerID="cri-o://609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4" gracePeriod=45
Jul 24 18:24:22.802579 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.802557    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:24:22.916251 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916217    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfrlf\" (UniqueName: \"kubernetes.io/projected/028d91c9-18c0-4f77-99f1-d46afee8282e-kube-api-access-nfrlf\") pod \"028d91c9-18c0-4f77-99f1-d46afee8282e\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") "
Jul 24 18:24:22.916444 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916275    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-home\") pod \"028d91c9-18c0-4f77-99f1-d46afee8282e\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") "
Jul 24 18:24:22.916444 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916336    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-model-cache\") pod \"028d91c9-18c0-4f77-99f1-d46afee8282e\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") "
Jul 24 18:24:22.916444 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916388    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-tmp-dir\") pod \"028d91c9-18c0-4f77-99f1-d46afee8282e\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") "
Jul 24 18:24:22.916590 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916445    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-dshm\") pod \"028d91c9-18c0-4f77-99f1-d46afee8282e\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") "
Jul 24 18:24:22.916590 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916487    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028d91c9-18c0-4f77-99f1-d46afee8282e-tls-certs\") pod \"028d91c9-18c0-4f77-99f1-d46afee8282e\" (UID: \"028d91c9-18c0-4f77-99f1-d46afee8282e\") "
Jul 24 18:24:22.916680 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916636    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-home" (OuterVolumeSpecName: "home") pod "028d91c9-18c0-4f77-99f1-d46afee8282e" (UID: "028d91c9-18c0-4f77-99f1-d46afee8282e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:22.916719 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916676    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "028d91c9-18c0-4f77-99f1-d46afee8282e" (UID: "028d91c9-18c0-4f77-99f1-d46afee8282e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:22.916784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916766    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-model-cache" (OuterVolumeSpecName: "model-cache") pod "028d91c9-18c0-4f77-99f1-d46afee8282e" (UID: "028d91c9-18c0-4f77-99f1-d46afee8282e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:22.916881 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916866    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.916940 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916883    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.916940 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.916892    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:22.918830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.918805    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028d91c9-18c0-4f77-99f1-d46afee8282e-kube-api-access-nfrlf" (OuterVolumeSpecName: "kube-api-access-nfrlf") pod "028d91c9-18c0-4f77-99f1-d46afee8282e" (UID: "028d91c9-18c0-4f77-99f1-d46afee8282e"). InnerVolumeSpecName "kube-api-access-nfrlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:24:22.918961 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.918941    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028d91c9-18c0-4f77-99f1-d46afee8282e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "028d91c9-18c0-4f77-99f1-d46afee8282e" (UID: "028d91c9-18c0-4f77-99f1-d46afee8282e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:24:22.919031 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:22.918993    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-dshm" (OuterVolumeSpecName: "dshm") pod "028d91c9-18c0-4f77-99f1-d46afee8282e" (UID: "028d91c9-18c0-4f77-99f1-d46afee8282e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:24:23.017837 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.017795    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028d91c9-18c0-4f77-99f1-d46afee8282e-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:23.017837 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.017828    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028d91c9-18c0-4f77-99f1-d46afee8282e-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:23.017837 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.017844    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrlf\" (UniqueName: \"kubernetes.io/projected/028d91c9-18c0-4f77-99f1-d46afee8282e-kube-api-access-nfrlf\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:24:23.260129 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.260035    2551 generic.go:356] "Generic (PLEG): container finished" podID="028d91c9-18c0-4f77-99f1-d46afee8282e" containerID="609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4" exitCode=0
Jul 24 18:24:23.260129 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.260074    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"
Jul 24 18:24:23.260129 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.260117    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx" event={"ID":"028d91c9-18c0-4f77-99f1-d46afee8282e","Type":"ContainerDied","Data":"609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4"}
Jul 24 18:24:23.260441 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.260153    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx" event={"ID":"028d91c9-18c0-4f77-99f1-d46afee8282e","Type":"ContainerDied","Data":"26e5a592c15f537e63051e528110f237220001f8ff64b408c4d15b1522f348f2"}
Jul 24 18:24:23.260441 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.260170    2551 scope.go:117] "RemoveContainer" containerID="609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4"
Jul 24 18:24:23.262107 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.262080    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr" event={"ID":"f65bf21c-a430-468a-a999-0773d5e41135","Type":"ContainerStarted","Data":"f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09"}
Jul 24 18:24:23.270573 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.270549    2551 scope.go:117] "RemoveContainer" containerID="609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4"
Jul 24 18:24:23.270856 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:24:23.270837    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4\": container with ID starting with 609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4 not found: ID does not exist" containerID="609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4"
Jul 24 18:24:23.270928 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.270865    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4"} err="failed to get container status \"609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4\": rpc error: code = NotFound desc = could not find container \"609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4\": container with ID starting with 609e6b74102f574f31c241142c31b4b3d4683707ff5e14f981cab63bf8054bc4 not found: ID does not exist"
Jul 24 18:24:23.282095 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.282032    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr" podStartSLOduration=2.28201813 podStartE2EDuration="2.28201813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:24:23.280816617 +0000 UTC m=+2654.397743981" watchObservedRunningTime="2026-07-24 18:24:23.28201813 +0000 UTC m=+2654.398945468"
Jul 24 18:24:23.293550 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.293526    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"]
Jul 24 18:24:23.302884 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.302861    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a379665a-f9f8-4f26-9e7a-c57099db98e2" path="/var/lib/kubelet/pods/a379665a-f9f8-4f26-9e7a-c57099db98e2/volumes"
Jul 24 18:24:23.303141 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:23.303129    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/auth-test-model-88d4a7-kserve-5595cc6687-mlqqx"]
Jul 24 18:24:24.156014 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:24.155970    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:24:25.300585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:25.300553    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="028d91c9-18c0-4f77-99f1-d46afee8282e" path="/var/lib/kubelet/pods/028d91c9-18c0-4f77-99f1-d46afee8282e/volumes"
Jul 24 18:24:32.055556 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:32.055524    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:32.056059 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:32.055572    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:32.068874 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:32.068844    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:24:32.323923 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:24:32.323834    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:25:08.507704 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.507628    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"]
Jul 24 18:25:08.508159 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.508108    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a379665a-f9f8-4f26-9e7a-c57099db98e2" containerName="main"
Jul 24 18:25:08.508159 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.508147    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a379665a-f9f8-4f26-9e7a-c57099db98e2" containerName="main"
Jul 24 18:25:08.508159 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.508155    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="028d91c9-18c0-4f77-99f1-d46afee8282e" containerName="main"
Jul 24 18:25:08.508159 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.508161    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="028d91c9-18c0-4f77-99f1-d46afee8282e" containerName="main"
Jul 24 18:25:08.508292 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.508236    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="028d91c9-18c0-4f77-99f1-d46afee8282e" containerName="main"
Jul 24 18:25:08.508292 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.508248    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a379665a-f9f8-4f26-9e7a-c57099db98e2" containerName="main"
Jul 24 18:25:08.511635 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.511611    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.514161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.514142    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"kube-root-ca.crt\""
Jul 24 18:25:08.515316 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.515296    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"default-dockercfg-lvjgj\""
Jul 24 18:25:08.515434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.515329    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"openshift-service-ca.crt\""
Jul 24 18:25:08.515434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.515328    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"sub-test-model-88d4a7-kserve-self-signed-certs\""
Jul 24 18:25:08.525333 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.525297    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"]
Jul 24 18:25:08.572039 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.572001    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-model-cache\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.572039 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.572041    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzw2\" (UniqueName: \"kubernetes.io/projected/9799c65f-1dc5-4a12-a51a-40f3e88003a9-kube-api-access-6mzw2\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.572284 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.572092    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tls-certs\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.572284 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.572134    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tmp-dir\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.572284 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.572187    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-dshm\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.572284 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.572256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-home\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.673850 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.673814    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-model-cache\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.673850 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.673852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzw2\" (UniqueName: \"kubernetes.io/projected/9799c65f-1dc5-4a12-a51a-40f3e88003a9-kube-api-access-6mzw2\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.674064 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.673888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tls-certs\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.674102 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.674063    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tmp-dir\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.674144 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.674105    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-dshm\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.674216 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.674199    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-home\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.674277 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.674232    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-model-cache\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.674436 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.674366    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tmp-dir\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.674577 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.674558    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-home\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.676474 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.676452    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-dshm\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.676576 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.676481    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tls-certs\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.682724 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.682703    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzw2\" (UniqueName: \"kubernetes.io/projected/9799c65f-1dc5-4a12-a51a-40f3e88003a9-kube-api-access-6mzw2\") pod \"sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") " pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.822972 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.822937    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:08.972883 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:08.972858    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"]
Jul 24 18:25:08.975751 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:25:08.975719    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9799c65f_1dc5_4a12_a51a_40f3e88003a9.slice/crio-8407a26fb941d43defdbe44939fefd4334f4082cda3d94c565caa61b8a963546 WatchSource:0}: Error finding container 8407a26fb941d43defdbe44939fefd4334f4082cda3d94c565caa61b8a963546: Status 404 returned error can't find the container with id 8407a26fb941d43defdbe44939fefd4334f4082cda3d94c565caa61b8a963546
Jul 24 18:25:09.451315 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:09.451289    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:25:09.460186 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:09.460167    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:25:09.511619 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:09.511582    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg" event={"ID":"9799c65f-1dc5-4a12-a51a-40f3e88003a9","Type":"ContainerStarted","Data":"c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8"}
Jul 24 18:25:09.511619 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:09.511626    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg" event={"ID":"9799c65f-1dc5-4a12-a51a-40f3e88003a9","Type":"ContainerStarted","Data":"8407a26fb941d43defdbe44939fefd4334f4082cda3d94c565caa61b8a963546"}
Jul 24 18:25:09.528711 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:09.528652    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg" podStartSLOduration=1.528636616 podStartE2EDuration="1.528636616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:25:09.526935744 +0000 UTC m=+2700.643863084" watchObservedRunningTime="2026-07-24 18:25:09.528636616 +0000 UTC m=+2700.645563987"
Jul 24 18:25:12.254709 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:12.254677    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:25:18.824042 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:18.823991    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:18.824505 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:18.824057    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:18.837392 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:18.837356    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:19.577401 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:19.577352    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:25:27.045193 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:27.045158    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:25:48.047920 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:48.047885    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:25:54.472767 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:54.472726    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"]
Jul 24 18:25:55.388793 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:25:55.388758    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"]
Jul 24 18:26:06.111784 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.111744    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"]
Jul 24 18:26:06.114255 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.114229    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.116899 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.116875    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b90928\"/\"rate-test-model-b90928-kserve-self-signed-certs\""
Jul 24 18:26:06.128133 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.128111    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"]
Jul 24 18:26:06.153762 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.153724    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tmp-dir\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.153942 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.153774    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-home\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.153942 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.153820    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tls-certs\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.153942 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.153857    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-dshm\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.153942 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.153927    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-model-cache\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.154136 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.153966    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltjrr\" (UniqueName: \"kubernetes.io/projected/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-kube-api-access-ltjrr\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255350 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-dshm\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255654 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255412    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-model-cache\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255654 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255445    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltjrr\" (UniqueName: \"kubernetes.io/projected/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-kube-api-access-ltjrr\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255654 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255508    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tmp-dir\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255654 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255549    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-home\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255654 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255598    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tls-certs\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255944 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255842    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-model-cache\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255944 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255922    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-home\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.255944 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.255937    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tmp-dir\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.257757 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.257733    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-dshm\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.258057 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.258037    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tls-certs\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.275533 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.275505    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltjrr\" (UniqueName: \"kubernetes.io/projected/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-kube-api-access-ltjrr\") pod \"rate-test-model-b90928-kserve-7886df47dc-jsv2p\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") " pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.425863 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.425776    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:06.567638 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.567608    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"]
Jul 24 18:26:06.570140 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:26:06.570111    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eccc331_efaf_4dc4_9add_48fb62cfbc0e.slice/crio-31b12e1d3ecf1de9556c15965d87316a0dd46e49f7ecf4df8cc616dc3c9d7e9f WatchSource:0}: Error finding container 31b12e1d3ecf1de9556c15965d87316a0dd46e49f7ecf4df8cc616dc3c9d7e9f: Status 404 returned error can't find the container with id 31b12e1d3ecf1de9556c15965d87316a0dd46e49f7ecf4df8cc616dc3c9d7e9f
Jul 24 18:26:06.814124 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.814085    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p" event={"ID":"7eccc331-efaf-4dc4-9add-48fb62cfbc0e","Type":"ContainerStarted","Data":"2e6dfca4940711ed8db6989a35b62eb1bd4988a4ee8f49e782ca5b5b83362eb3"}
Jul 24 18:26:06.814299 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.814132    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p" event={"ID":"7eccc331-efaf-4dc4-9add-48fb62cfbc0e","Type":"ContainerStarted","Data":"31b12e1d3ecf1de9556c15965d87316a0dd46e49f7ecf4df8cc616dc3c9d7e9f"}
Jul 24 18:26:06.834221 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:06.834170    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p" podStartSLOduration=0.834156488 podStartE2EDuration="834.156488ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:26:06.832463578 +0000 UTC m=+2757.949390921" watchObservedRunningTime="2026-07-24 18:26:06.834156488 +0000 UTC m=+2757.951083827"
Jul 24 18:26:09.147787 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.147749    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:26:09.493328 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.493220    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr" podUID="f65bf21c-a430-468a-a999-0773d5e41135" containerName="main" containerID="cri-o://f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09" gracePeriod=45
Jul 24 18:26:09.748593 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.748539    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:26:09.791611 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791579    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-dshm\") pod \"f65bf21c-a430-468a-a999-0773d5e41135\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") "
Jul 24 18:26:09.791756 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791632    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-tmp-dir\") pod \"f65bf21c-a430-468a-a999-0773d5e41135\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") "
Jul 24 18:26:09.791756 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791680    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdxmb\" (UniqueName: \"kubernetes.io/projected/f65bf21c-a430-468a-a999-0773d5e41135-kube-api-access-jdxmb\") pod \"f65bf21c-a430-468a-a999-0773d5e41135\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") "
Jul 24 18:26:09.791756 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791711    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-model-cache\") pod \"f65bf21c-a430-468a-a999-0773d5e41135\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") "
Jul 24 18:26:09.791878 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791843    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-home\") pod \"f65bf21c-a430-468a-a999-0773d5e41135\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") "
Jul 24 18:26:09.791948 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791925    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65bf21c-a430-468a-a999-0773d5e41135-tls-certs\") pod \"f65bf21c-a430-468a-a999-0773d5e41135\" (UID: \"f65bf21c-a430-468a-a999-0773d5e41135\") "
Jul 24 18:26:09.792008 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791962    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-model-cache" (OuterVolumeSpecName: "model-cache") pod "f65bf21c-a430-468a-a999-0773d5e41135" (UID: "f65bf21c-a430-468a-a999-0773d5e41135"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:09.792008 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.791981    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f65bf21c-a430-468a-a999-0773d5e41135" (UID: "f65bf21c-a430-468a-a999-0773d5e41135"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:09.792153 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.792120    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-home" (OuterVolumeSpecName: "home") pod "f65bf21c-a430-468a-a999-0773d5e41135" (UID: "f65bf21c-a430-468a-a999-0773d5e41135"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:09.792337 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.792313    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:09.792444 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.792342    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:09.792444 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.792356    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:09.794002 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.793979    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-dshm" (OuterVolumeSpecName: "dshm") pod "f65bf21c-a430-468a-a999-0773d5e41135" (UID: "f65bf21c-a430-468a-a999-0773d5e41135"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:09.794266 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.794249    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65bf21c-a430-468a-a999-0773d5e41135-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f65bf21c-a430-468a-a999-0773d5e41135" (UID: "f65bf21c-a430-468a-a999-0773d5e41135"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:26:09.794266 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.794248    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65bf21c-a430-468a-a999-0773d5e41135-kube-api-access-jdxmb" (OuterVolumeSpecName: "kube-api-access-jdxmb") pod "f65bf21c-a430-468a-a999-0773d5e41135" (UID: "f65bf21c-a430-468a-a999-0773d5e41135"). InnerVolumeSpecName "kube-api-access-jdxmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:26:09.832120 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.832089    2551 generic.go:356] "Generic (PLEG): container finished" podID="f65bf21c-a430-468a-a999-0773d5e41135" containerID="f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09" exitCode=0
Jul 24 18:26:09.832283 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.832134    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"
Jul 24 18:26:09.832283 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.832172    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr" event={"ID":"f65bf21c-a430-468a-a999-0773d5e41135","Type":"ContainerDied","Data":"f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09"}
Jul 24 18:26:09.832283 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.832203    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr" event={"ID":"f65bf21c-a430-468a-a999-0773d5e41135","Type":"ContainerDied","Data":"ba45a9cdcaa4cad40c7968a719c4c5f016631d1bd212d68c1383c783c15f3546"}
Jul 24 18:26:09.832283 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.832220    2551 scope.go:117] "RemoveContainer" containerID="f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09"
Jul 24 18:26:09.841350 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.841334    2551 scope.go:117] "RemoveContainer" containerID="f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09"
Jul 24 18:26:09.841623 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:26:09.841605    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09\": container with ID starting with f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09 not found: ID does not exist" containerID="f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09"
Jul 24 18:26:09.841671 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.841632    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09"} err="failed to get container status \"f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09\": rpc error: code = NotFound desc = could not find container \"f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09\": container with ID starting with f80a3eda935672104e90e10e0d8ef1d78a6c1f0cf9a268f83b332f5abcc84d09 not found: ID does not exist"
Jul 24 18:26:09.854843 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.854818    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"]
Jul 24 18:26:09.858689 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.858669    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/sub-test-model-b90928-kserve-86d4b58fb7-rlpcr"]
Jul 24 18:26:09.893142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.893119    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f65bf21c-a430-468a-a999-0773d5e41135-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:09.893142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.893141    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f65bf21c-a430-468a-a999-0773d5e41135-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:09.893329 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:09.893154    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxmb\" (UniqueName: \"kubernetes.io/projected/f65bf21c-a430-468a-a999-0773d5e41135-kube-api-access-jdxmb\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:10.410094 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.410055    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg" podUID="9799c65f-1dc5-4a12-a51a-40f3e88003a9" containerName="main" containerID="cri-o://c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8" gracePeriod=45
Jul 24 18:26:10.672205 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.672137    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:26:10.702972 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.702940    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-model-cache\") pod \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") "
Jul 24 18:26:10.703133 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.702986    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mzw2\" (UniqueName: \"kubernetes.io/projected/9799c65f-1dc5-4a12-a51a-40f3e88003a9-kube-api-access-6mzw2\") pod \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") "
Jul 24 18:26:10.703133 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703047    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tls-certs\") pod \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") "
Jul 24 18:26:10.703133 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703066    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-home\") pod \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") "
Jul 24 18:26:10.703133 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703111    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tmp-dir\") pod \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") "
Jul 24 18:26:10.703360 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703189    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-dshm\") pod \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\" (UID: \"9799c65f-1dc5-4a12-a51a-40f3e88003a9\") "
Jul 24 18:26:10.703360 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703187    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9799c65f-1dc5-4a12-a51a-40f3e88003a9" (UID: "9799c65f-1dc5-4a12-a51a-40f3e88003a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:10.703513 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703441    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-home" (OuterVolumeSpecName: "home") pod "9799c65f-1dc5-4a12-a51a-40f3e88003a9" (UID: "9799c65f-1dc5-4a12-a51a-40f3e88003a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:10.703513 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703473    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9799c65f-1dc5-4a12-a51a-40f3e88003a9" (UID: "9799c65f-1dc5-4a12-a51a-40f3e88003a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:10.703653 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703633    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:10.703710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703661    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:10.703710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.703675    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:10.705344 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.705311    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9799c65f-1dc5-4a12-a51a-40f3e88003a9" (UID: "9799c65f-1dc5-4a12-a51a-40f3e88003a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:26:10.705344 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.705311    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-dshm" (OuterVolumeSpecName: "dshm") pod "9799c65f-1dc5-4a12-a51a-40f3e88003a9" (UID: "9799c65f-1dc5-4a12-a51a-40f3e88003a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:26:10.705745 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.705722    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9799c65f-1dc5-4a12-a51a-40f3e88003a9-kube-api-access-6mzw2" (OuterVolumeSpecName: "kube-api-access-6mzw2") pod "9799c65f-1dc5-4a12-a51a-40f3e88003a9" (UID: "9799c65f-1dc5-4a12-a51a-40f3e88003a9"). InnerVolumeSpecName "kube-api-access-6mzw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:26:10.805084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.805053    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9799c65f-1dc5-4a12-a51a-40f3e88003a9-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:10.805084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.805080    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9799c65f-1dc5-4a12-a51a-40f3e88003a9-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:10.805084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.805090    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mzw2\" (UniqueName: \"kubernetes.io/projected/9799c65f-1dc5-4a12-a51a-40f3e88003a9-kube-api-access-6mzw2\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:26:10.839739 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.839705    2551 generic.go:356] "Generic (PLEG): container finished" podID="9799c65f-1dc5-4a12-a51a-40f3e88003a9" containerID="c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8" exitCode=0
Jul 24 18:26:10.839954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.839757    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"
Jul 24 18:26:10.839954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.839760    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg" event={"ID":"9799c65f-1dc5-4a12-a51a-40f3e88003a9","Type":"ContainerDied","Data":"c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8"}
Jul 24 18:26:10.839954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.839799    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg" event={"ID":"9799c65f-1dc5-4a12-a51a-40f3e88003a9","Type":"ContainerDied","Data":"8407a26fb941d43defdbe44939fefd4334f4082cda3d94c565caa61b8a963546"}
Jul 24 18:26:10.839954 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.839821    2551 scope.go:117] "RemoveContainer" containerID="c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8"
Jul 24 18:26:10.849416 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.849368    2551 scope.go:117] "RemoveContainer" containerID="c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8"
Jul 24 18:26:10.849708 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:26:10.849678    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8\": container with ID starting with c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8 not found: ID does not exist" containerID="c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8"
Jul 24 18:26:10.849785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.849718    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8"} err="failed to get container status \"c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8\": rpc error: code = NotFound desc = could not find container \"c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8\": container with ID starting with c7055f87b45897877712fafed6095812e48010c1b2737efcb16808aa817a8ed8 not found: ID does not exist"
Jul 24 18:26:10.862039 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.862015    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"]
Jul 24 18:26:10.866415 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:10.866368    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/sub-test-model-88d4a7-kserve-6d9cdc8f47-d4ldg"]
Jul 24 18:26:11.308671 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:11.308625    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9799c65f-1dc5-4a12-a51a-40f3e88003a9" path="/var/lib/kubelet/pods/9799c65f-1dc5-4a12-a51a-40f3e88003a9/volumes"
Jul 24 18:26:11.309084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:11.309070    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65bf21c-a430-468a-a999-0773d5e41135" path="/var/lib/kubelet/pods/f65bf21c-a430-468a-a999-0773d5e41135/volumes"
Jul 24 18:26:16.426234 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:16.426189    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:16.426761 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:16.426260    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:16.439287 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:16.439262    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:16.881324 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:16.881295    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:26:23.006288 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.006244    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"]
Jul 24 18:26:23.006949 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.006930    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65bf21c-a430-468a-a999-0773d5e41135" containerName="main"
Jul 24 18:26:23.006949 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.006949    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65bf21c-a430-468a-a999-0773d5e41135" containerName="main"
Jul 24 18:26:23.007079 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.006976    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9799c65f-1dc5-4a12-a51a-40f3e88003a9" containerName="main"
Jul 24 18:26:23.007079 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.006984    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9799c65f-1dc5-4a12-a51a-40f3e88003a9" containerName="main"
Jul 24 18:26:23.007182 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.007093    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65bf21c-a430-468a-a999-0773d5e41135" containerName="main"
Jul 24 18:26:23.007182 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.007109    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9799c65f-1dc5-4a12-a51a-40f3e88003a9" containerName="main"
Jul 24 18:26:23.010997 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.010975    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.014044 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.014022    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"rate-test-model-88d4a7-kserve-self-signed-certs\""
Jul 24 18:26:23.014976 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.014959    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"default-dockercfg-lvjgj\""
Jul 24 18:26:23.015036 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.014979    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"kube-root-ca.crt\""
Jul 24 18:26:23.015085 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.014959    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88d4a7\"/\"openshift-service-ca.crt\""
Jul 24 18:26:23.020610 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.020570    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"]
Jul 24 18:26:23.130657 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.130617    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tmp-dir\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.130845 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.130682    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tls-certs\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.130845 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.130701    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9lkl\" (UniqueName: \"kubernetes.io/projected/debcf4b0-1190-4a4f-a548-eeacc39ab11c-kube-api-access-n9lkl\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.130845 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.130745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-model-cache\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.130845 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.130798    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-home\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.130845 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.130841    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-dshm\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232462 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232422    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-dshm\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tmp-dir\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232568    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tls-certs\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232586    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9lkl\" (UniqueName: \"kubernetes.io/projected/debcf4b0-1190-4a4f-a548-eeacc39ab11c-kube-api-access-n9lkl\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232604    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-model-cache\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232626    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-home\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232967 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232941    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tmp-dir\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.232967 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232956    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-model-cache\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.233063 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.232997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-home\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.234991 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.234966    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-dshm\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.235266 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.235246    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tls-certs\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.241409 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.241368    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9lkl\" (UniqueName: \"kubernetes.io/projected/debcf4b0-1190-4a4f-a548-eeacc39ab11c-kube-api-access-n9lkl\") pod \"rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") " pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.322206 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.322173    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:23.463733 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.463702    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"]
Jul 24 18:26:23.465766 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:26:23.465739    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddebcf4b0_1190_4a4f_a548_eeacc39ab11c.slice/crio-33c3ab67e825103637a9fdd13d068b026006b04adde4429a23592fcc8f629a10 WatchSource:0}: Error finding container 33c3ab67e825103637a9fdd13d068b026006b04adde4429a23592fcc8f629a10: Status 404 returned error can't find the container with id 33c3ab67e825103637a9fdd13d068b026006b04adde4429a23592fcc8f629a10
Jul 24 18:26:23.907797 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.907758    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz" event={"ID":"debcf4b0-1190-4a4f-a548-eeacc39ab11c","Type":"ContainerStarted","Data":"41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429"}
Jul 24 18:26:23.907994 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.907808    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz" event={"ID":"debcf4b0-1190-4a4f-a548-eeacc39ab11c","Type":"ContainerStarted","Data":"33c3ab67e825103637a9fdd13d068b026006b04adde4429a23592fcc8f629a10"}
Jul 24 18:26:23.945424 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:23.945330    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz" podStartSLOduration=1.945308776 podStartE2EDuration="1.945308776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:26:23.944447545 +0000 UTC m=+2775.061374884" watchObservedRunningTime="2026-07-24 18:26:23.945308776 +0000 UTC m=+2775.062236119"
Jul 24 18:26:26.769244 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:26.769205    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:26:33.322359 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:33.322282    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:33.322779 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:33.322447    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:33.335688 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:33.335650    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:33.971521 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:33.971489    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:26:52.345006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:26:52.344969    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:27:08.455024 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:08.454993    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:27:15.003151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:15.003114    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"]
Jul 24 18:27:15.991161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:15.991116    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"]
Jul 24 18:27:23.450614 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:23.450574    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:27:30.022756 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.022689    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p" podUID="7eccc331-efaf-4dc4-9add-48fb62cfbc0e" containerName="main" containerID="cri-o://2e6dfca4940711ed8db6989a35b62eb1bd4988a4ee8f49e782ca5b5b83362eb3" gracePeriod=45
Jul 24 18:27:30.265156 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.265117    2551 generic.go:356] "Generic (PLEG): container finished" podID="7eccc331-efaf-4dc4-9add-48fb62cfbc0e" containerID="2e6dfca4940711ed8db6989a35b62eb1bd4988a4ee8f49e782ca5b5b83362eb3" exitCode=0
Jul 24 18:27:30.265409 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.265193    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p" event={"ID":"7eccc331-efaf-4dc4-9add-48fb62cfbc0e","Type":"ContainerDied","Data":"2e6dfca4940711ed8db6989a35b62eb1bd4988a4ee8f49e782ca5b5b83362eb3"}
Jul 24 18:27:30.305538 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.305503    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:27:30.390197 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390161    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-home\") pod \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") "
Jul 24 18:27:30.390369 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390248    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-dshm\") pod \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") "
Jul 24 18:27:30.390369 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390287    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-model-cache\") pod \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") "
Jul 24 18:27:30.390369 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390332    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tmp-dir\") pod \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") "
Jul 24 18:27:30.390531 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390413    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tls-certs\") pod \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") "
Jul 24 18:27:30.390531 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390458    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltjrr\" (UniqueName: \"kubernetes.io/projected/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-kube-api-access-ltjrr\") pod \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\" (UID: \"7eccc331-efaf-4dc4-9add-48fb62cfbc0e\") "
Jul 24 18:27:30.390531 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390501    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-home" (OuterVolumeSpecName: "home") pod "7eccc331-efaf-4dc4-9add-48fb62cfbc0e" (UID: "7eccc331-efaf-4dc4-9add-48fb62cfbc0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:30.390679 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390635    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7eccc331-efaf-4dc4-9add-48fb62cfbc0e" (UID: "7eccc331-efaf-4dc4-9add-48fb62cfbc0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:30.390679 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390624    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7eccc331-efaf-4dc4-9add-48fb62cfbc0e" (UID: "7eccc331-efaf-4dc4-9add-48fb62cfbc0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:30.390835 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390812    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:30.390835 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390836    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:30.391004 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.390848    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:30.392772 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.392738    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-kube-api-access-ltjrr" (OuterVolumeSpecName: "kube-api-access-ltjrr") pod "7eccc331-efaf-4dc4-9add-48fb62cfbc0e" (UID: "7eccc331-efaf-4dc4-9add-48fb62cfbc0e"). InnerVolumeSpecName "kube-api-access-ltjrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:27:30.392772 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.392764    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7eccc331-efaf-4dc4-9add-48fb62cfbc0e" (UID: "7eccc331-efaf-4dc4-9add-48fb62cfbc0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:27:30.392933 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.392895    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-dshm" (OuterVolumeSpecName: "dshm") pod "7eccc331-efaf-4dc4-9add-48fb62cfbc0e" (UID: "7eccc331-efaf-4dc4-9add-48fb62cfbc0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:30.491623 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.491586    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:30.491623 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.491620    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:30.491831 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:30.491636    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltjrr\" (UniqueName: \"kubernetes.io/projected/7eccc331-efaf-4dc4-9add-48fb62cfbc0e-kube-api-access-ltjrr\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:31.010486 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.010445    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz" podUID="debcf4b0-1190-4a4f-a548-eeacc39ab11c" containerName="main" containerID="cri-o://41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429" gracePeriod=45
Jul 24 18:27:31.270712 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.270645    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:27:31.272305 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.272278    2551 generic.go:356] "Generic (PLEG): container finished" podID="debcf4b0-1190-4a4f-a548-eeacc39ab11c" containerID="41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429" exitCode=0
Jul 24 18:27:31.272439 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.272315    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"
Jul 24 18:27:31.272439 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.272353    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz" event={"ID":"debcf4b0-1190-4a4f-a548-eeacc39ab11c","Type":"ContainerDied","Data":"41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429"}
Jul 24 18:27:31.272439 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.272421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz" event={"ID":"debcf4b0-1190-4a4f-a548-eeacc39ab11c","Type":"ContainerDied","Data":"33c3ab67e825103637a9fdd13d068b026006b04adde4429a23592fcc8f629a10"}
Jul 24 18:27:31.272557 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.272442    2551 scope.go:117] "RemoveContainer" containerID="41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429"
Jul 24 18:27:31.273801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.273779    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"
Jul 24 18:27:31.273900 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.273802    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p" event={"ID":"7eccc331-efaf-4dc4-9add-48fb62cfbc0e","Type":"ContainerDied","Data":"31b12e1d3ecf1de9556c15965d87316a0dd46e49f7ecf4df8cc616dc3c9d7e9f"}
Jul 24 18:27:31.281844 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.281833    2551 scope.go:117] "RemoveContainer" containerID="41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429"
Jul 24 18:27:31.282169 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:27:31.282142    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429\": container with ID starting with 41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429 not found: ID does not exist" containerID="41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429"
Jul 24 18:27:31.282254 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.282180    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429"} err="failed to get container status \"41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429\": rpc error: code = NotFound desc = could not find container \"41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429\": container with ID starting with 41b0be978e2df977903b1726e052264010e2ca51b30195e41c3a088dd624d429 not found: ID does not exist"
Jul 24 18:27:31.282254 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.282204    2551 scope.go:117] "RemoveContainer" containerID="2e6dfca4940711ed8db6989a35b62eb1bd4988a4ee8f49e782ca5b5b83362eb3"
Jul 24 18:27:31.307874 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.307844    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"]
Jul 24 18:27:31.313514 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.313487    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b90928/rate-test-model-b90928-kserve-7886df47dc-jsv2p"]
Jul 24 18:27:31.401588 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401559    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tmp-dir\") pod \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") "
Jul 24 18:27:31.401773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401623    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-home\") pod \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") "
Jul 24 18:27:31.401773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401660    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-model-cache\") pod \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") "
Jul 24 18:27:31.401773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401685    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9lkl\" (UniqueName: \"kubernetes.io/projected/debcf4b0-1190-4a4f-a548-eeacc39ab11c-kube-api-access-n9lkl\") pod \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") "
Jul 24 18:27:31.401773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401753    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-dshm\") pod \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") "
Jul 24 18:27:31.401773 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401772    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tls-certs\") pod \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\" (UID: \"debcf4b0-1190-4a4f-a548-eeacc39ab11c\") "
Jul 24 18:27:31.401981 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401836    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "debcf4b0-1190-4a4f-a548-eeacc39ab11c" (UID: "debcf4b0-1190-4a4f-a548-eeacc39ab11c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:31.401981 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401944    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-home" (OuterVolumeSpecName: "home") pod "debcf4b0-1190-4a4f-a548-eeacc39ab11c" (UID: "debcf4b0-1190-4a4f-a548-eeacc39ab11c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:31.401981 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.401955    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-model-cache" (OuterVolumeSpecName: "model-cache") pod "debcf4b0-1190-4a4f-a548-eeacc39ab11c" (UID: "debcf4b0-1190-4a4f-a548-eeacc39ab11c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:31.402137 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.402119    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:31.402200 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.402138    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:31.402200 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.402147    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:31.404032 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.404002    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debcf4b0-1190-4a4f-a548-eeacc39ab11c-kube-api-access-n9lkl" (OuterVolumeSpecName: "kube-api-access-n9lkl") pod "debcf4b0-1190-4a4f-a548-eeacc39ab11c" (UID: "debcf4b0-1190-4a4f-a548-eeacc39ab11c"). InnerVolumeSpecName "kube-api-access-n9lkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:27:31.404147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.404032    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-dshm" (OuterVolumeSpecName: "dshm") pod "debcf4b0-1190-4a4f-a548-eeacc39ab11c" (UID: "debcf4b0-1190-4a4f-a548-eeacc39ab11c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:27:31.404147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.404035    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "debcf4b0-1190-4a4f-a548-eeacc39ab11c" (UID: "debcf4b0-1190-4a4f-a548-eeacc39ab11c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:27:31.503581 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.503544    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9lkl\" (UniqueName: \"kubernetes.io/projected/debcf4b0-1190-4a4f-a548-eeacc39ab11c-kube-api-access-n9lkl\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:31.503581 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.503572    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debcf4b0-1190-4a4f-a548-eeacc39ab11c-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:31.503581 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.503581    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debcf4b0-1190-4a4f-a548-eeacc39ab11c-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:27:31.598734 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.598696    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"]
Jul 24 18:27:31.601961 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:31.601932    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-88d4a7/rate-test-model-88d4a7-kserve-6dcd95695f-x5vdz"]
Jul 24 18:27:33.300558 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:33.300525    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eccc331-efaf-4dc4-9add-48fb62cfbc0e" path="/var/lib/kubelet/pods/7eccc331-efaf-4dc4-9add-48fb62cfbc0e/volumes"
Jul 24 18:27:33.300937 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:33.300927    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="debcf4b0-1190-4a4f-a548-eeacc39ab11c" path="/var/lib/kubelet/pods/debcf4b0-1190-4a4f-a548-eeacc39ab11c/volumes"
Jul 24 18:27:40.737855 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.737822    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"]
Jul 24 18:27:40.738332 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.738236    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="debcf4b0-1190-4a4f-a548-eeacc39ab11c" containerName="main"
Jul 24 18:27:40.738332 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.738248    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="debcf4b0-1190-4a4f-a548-eeacc39ab11c" containerName="main"
Jul 24 18:27:40.738332 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.738267    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eccc331-efaf-4dc4-9add-48fb62cfbc0e" containerName="main"
Jul 24 18:27:40.738332 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.738273    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eccc331-efaf-4dc4-9add-48fb62cfbc0e" containerName="main"
Jul 24 18:27:40.738492 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.738357    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="debcf4b0-1190-4a4f-a548-eeacc39ab11c" containerName="main"
Jul 24 18:27:40.738492 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.738370    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eccc331-efaf-4dc4-9add-48fb62cfbc0e" containerName="main"
Jul 24 18:27:40.741414 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.741396    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:40.743781 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.743758    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-378627\""
Jul 24 18:27:40.743781 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.743770    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-378627-dockercfg-hq4dp\""
Jul 24 18:27:40.754444 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.754424    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"]
Jul 24 18:27:40.767334 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.767308    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"]
Jul 24 18:27:40.771133 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.771111    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:40.781700 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.781676    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"]
Jul 24 18:27:40.895574 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.895527    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88j5\" (UniqueName: \"kubernetes.io/projected/9b8f1899-cb68-4877-b4e0-1226638bcd78-kube-api-access-k88j5\") pod \"payload-processing-e2e-ipp-a-378627-746d9d945b-sh584\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:40.895574 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.895575    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8f1899-cb68-4877-b4e0-1226638bcd78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-378627-746d9d945b-sh584\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:40.895857 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.895593    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61308c9e-cae8-4e33-b17a-f7604213fad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:40.895857 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.895712    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szph5\" (UniqueName: \"kubernetes.io/projected/61308c9e-cae8-4e33-b17a-f7604213fad4-kube-api-access-szph5\") pod \"payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:40.996971 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.996866    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88j5\" (UniqueName: \"kubernetes.io/projected/9b8f1899-cb68-4877-b4e0-1226638bcd78-kube-api-access-k88j5\") pod \"payload-processing-e2e-ipp-a-378627-746d9d945b-sh584\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:40.996971 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.996928    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8f1899-cb68-4877-b4e0-1226638bcd78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-378627-746d9d945b-sh584\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:40.997223 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.997059    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61308c9e-cae8-4e33-b17a-f7604213fad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:40.997223 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.997199    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szph5\" (UniqueName: \"kubernetes.io/projected/61308c9e-cae8-4e33-b17a-f7604213fad4-kube-api-access-szph5\") pod \"payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:40.997841 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.997815    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8f1899-cb68-4877-b4e0-1226638bcd78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-378627-746d9d945b-sh584\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:40.998035 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:40.998012    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61308c9e-cae8-4e33-b17a-f7604213fad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:41.004997 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.004970    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88j5\" (UniqueName: \"kubernetes.io/projected/9b8f1899-cb68-4877-b4e0-1226638bcd78-kube-api-access-k88j5\") pod \"payload-processing-e2e-ipp-a-378627-746d9d945b-sh584\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:41.005138 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.005117    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szph5\" (UniqueName: \"kubernetes.io/projected/61308c9e-cae8-4e33-b17a-f7604213fad4-kube-api-access-szph5\") pod \"payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:41.051992 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.051960    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:41.084161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.084132    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:41.218093 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.218062    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"]
Jul 24 18:27:41.220396 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:27:41.220340    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61308c9e_cae8_4e33_b17a_f7604213fad4.slice/crio-baa8e4c2dfff376765040e415aa88afbe2261831dfbc361b3ab724b3cd69a1f9 WatchSource:0}: Error finding container baa8e4c2dfff376765040e415aa88afbe2261831dfbc361b3ab724b3cd69a1f9: Status 404 returned error can't find the container with id baa8e4c2dfff376765040e415aa88afbe2261831dfbc361b3ab724b3cd69a1f9
Jul 24 18:27:41.248778 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.248707    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"]
Jul 24 18:27:41.258661 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:27:41.258636    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8f1899_cb68_4877_b4e0_1226638bcd78.slice/crio-a1636903fd357220f94025f8283dd9ac7c6d44a59c6f28f3859882bb051c9e1c WatchSource:0}: Error finding container a1636903fd357220f94025f8283dd9ac7c6d44a59c6f28f3859882bb051c9e1c: Status 404 returned error can't find the container with id a1636903fd357220f94025f8283dd9ac7c6d44a59c6f28f3859882bb051c9e1c
Jul 24 18:27:41.329493 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.329438    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584" event={"ID":"9b8f1899-cb68-4877-b4e0-1226638bcd78","Type":"ContainerStarted","Data":"9cc3b88a832187e085c17aa0a9187491a39afc7f8a2271d244543757e2010fe1"}
Jul 24 18:27:41.329712 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.329507    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584" event={"ID":"9b8f1899-cb68-4877-b4e0-1226638bcd78","Type":"ContainerStarted","Data":"a1636903fd357220f94025f8283dd9ac7c6d44a59c6f28f3859882bb051c9e1c"}
Jul 24 18:27:41.329712 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.329616    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:41.331285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.331253    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v" event={"ID":"61308c9e-cae8-4e33-b17a-f7604213fad4","Type":"ContainerStarted","Data":"1a574dd04e1523d8ef3b4780c8d17b583121f5a880d7cf0d052f089bd1a6ccae"}
Jul 24 18:27:41.331452 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.331306    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v" event={"ID":"61308c9e-cae8-4e33-b17a-f7604213fad4","Type":"ContainerStarted","Data":"baa8e4c2dfff376765040e415aa88afbe2261831dfbc361b3ab724b3cd69a1f9"}
Jul 24 18:27:41.331452 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.331360    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:41.365347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.365287    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v" podStartSLOduration=1.365271068 podStartE2EDuration="1.365271068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:27:41.364353602 +0000 UTC m=+2852.481280943" watchObservedRunningTime="2026-07-24 18:27:41.365271068 +0000 UTC m=+2852.482198408"
Jul 24 18:27:41.365583 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:41.365402    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584" podStartSLOduration=1.365393819 podStartE2EDuration="1.365393819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:27:41.346597909 +0000 UTC m=+2852.463525256" watchObservedRunningTime="2026-07-24 18:27:41.365393819 +0000 UTC m=+2852.482321161"
Jul 24 18:27:48.249346 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:48.249309    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:27:52.338525 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:52.338482    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:27:52.338919 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:52.338771    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:27:59.682688 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.682597    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"]
Jul 24 18:27:59.686528 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.686502    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:27:59.689006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.688984    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cb47da\""
Jul 24 18:27:59.689135 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.689012    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cb47da-dockercfg-2xvbj\""
Jul 24 18:27:59.698689 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.698660    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"]
Jul 24 18:27:59.717575 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.717549    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"]
Jul 24 18:27:59.721340 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.721322    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:27:59.734708 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.734682    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"]
Jul 24 18:27:59.782649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.782612    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:27:59.782837 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.782688    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtnl\" (UniqueName: \"kubernetes.io/projected/999889f7-5d6e-4398-b843-1cea4eeda865-kube-api-access-pmtnl\") pod \"payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:27:59.782837 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.782749    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999889f7-5d6e-4398-b843-1cea4eeda865-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:27:59.782837 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.782800    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fh6\" (UniqueName: \"kubernetes.io/projected/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-kube-api-access-g8fh6\") pod \"payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:27:59.883695 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.883646    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtnl\" (UniqueName: \"kubernetes.io/projected/999889f7-5d6e-4398-b843-1cea4eeda865-kube-api-access-pmtnl\") pod \"payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:27:59.883695 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.883704    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999889f7-5d6e-4398-b843-1cea4eeda865-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:27:59.883971 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.883730    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fh6\" (UniqueName: \"kubernetes.io/projected/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-kube-api-access-g8fh6\") pod \"payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:27:59.883971 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.883777    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:27:59.884479 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.884459    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999889f7-5d6e-4398-b843-1cea4eeda865-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:27:59.884530 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.884488    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:27:59.894317 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.894288    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtnl\" (UniqueName: \"kubernetes.io/projected/999889f7-5d6e-4398-b843-1cea4eeda865-kube-api-access-pmtnl\") pod \"payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:27:59.894905 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.894878    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fh6\" (UniqueName: \"kubernetes.io/projected/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-kube-api-access-g8fh6\") pod \"payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:27:59.999282 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:27:59.999188    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:28:00.031885 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.031847    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:28:00.166882 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.166769    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"]
Jul 24 18:28:00.171029 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:28:00.170992    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454dbfc7_61bd_4371_b3c3_524d8d7b6a55.slice/crio-4df596d462bd12008f8c72b62039d39f40844b758e96b8b07db74220123a56e9 WatchSource:0}: Error finding container 4df596d462bd12008f8c72b62039d39f40844b758e96b8b07db74220123a56e9: Status 404 returned error can't find the container with id 4df596d462bd12008f8c72b62039d39f40844b758e96b8b07db74220123a56e9
Jul 24 18:28:00.398862 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.398834    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"]
Jul 24 18:28:00.401007 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:28:00.400976    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999889f7_5d6e_4398_b843_1cea4eeda865.slice/crio-7415c814dfe338807b2ee9fbf729fd62ab4b58fc33fe75be2b1da2a11b1091d5 WatchSource:0}: Error finding container 7415c814dfe338807b2ee9fbf729fd62ab4b58fc33fe75be2b1da2a11b1091d5: Status 404 returned error can't find the container with id 7415c814dfe338807b2ee9fbf729fd62ab4b58fc33fe75be2b1da2a11b1091d5
Jul 24 18:28:00.436081 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.436051    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b" event={"ID":"454dbfc7-61bd-4371-b3c3-524d8d7b6a55","Type":"ContainerStarted","Data":"e5c52f4a45dcc564c3f96d7086c26de909c8a76dab46ec478ffe099eb773b879"}
Jul 24 18:28:00.436231 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.436110    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b" event={"ID":"454dbfc7-61bd-4371-b3c3-524d8d7b6a55","Type":"ContainerStarted","Data":"4df596d462bd12008f8c72b62039d39f40844b758e96b8b07db74220123a56e9"}
Jul 24 18:28:00.436231 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.436151    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:28:00.437520 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.437499    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r" event={"ID":"999889f7-5d6e-4398-b843-1cea4eeda865","Type":"ContainerStarted","Data":"7415c814dfe338807b2ee9fbf729fd62ab4b58fc33fe75be2b1da2a11b1091d5"}
Jul 24 18:28:00.453415 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:00.453353    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b" podStartSLOduration=1.453336671 podStartE2EDuration="1.453336671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:28:00.452546371 +0000 UTC m=+2871.569473712" watchObservedRunningTime="2026-07-24 18:28:00.453336671 +0000 UTC m=+2871.570264008"
Jul 24 18:28:01.444820 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:01.444783    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r" event={"ID":"999889f7-5d6e-4398-b843-1cea4eeda865","Type":"ContainerStarted","Data":"1046afdf05342918fc09e68ad31940a9b351de08c0c1d74618bcc63545d54d18"}
Jul 24 18:28:01.445275 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:01.444860    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:28:07.240343 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:07.240286    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r" podStartSLOduration=8.240268483 podStartE2EDuration="8.240268483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:28:01.502595437 +0000 UTC m=+2872.619522777" watchObservedRunningTime="2026-07-24 18:28:07.240268483 +0000 UTC m=+2878.357195844"
Jul 24 18:28:07.241530 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:07.241510    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:28:11.447285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:11.447252    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:28:12.451934 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:12.451901    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:28:19.490887 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:19.490857    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b_454dbfc7-61bd-4371-b3c3-524d8d7b6a55/payload-processing/0.log"
Jul 24 18:28:36.310327 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.310282    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"]
Jul 24 18:28:36.314571 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.314549    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.316977 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.316954    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-378627\"/\"openshift-service-ca.crt\""
Jul 24 18:28:36.317095 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.316956    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-378627\"/\"kube-root-ca.crt\""
Jul 24 18:28:36.317741 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.317724    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-378627\"/\"default-dockercfg-444kj\""
Jul 24 18:28:36.317801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.317760    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-378627\"/\"ipp-route-378627-kserve-self-signed-certs\""
Jul 24 18:28:36.323541 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.323515    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"]
Jul 24 18:28:36.437353 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.437307    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzks\" (UniqueName: \"kubernetes.io/projected/1fdda175-7281-4321-8554-d38310c31715-kube-api-access-xbzks\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.437599 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.437360    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-model-cache\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.437599 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.437518    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-home\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.437599 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.437591    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdda175-7281-4321-8554-d38310c31715-tls-certs\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.437787 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.437638    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-dshm\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.437787 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.437777    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-tmp-dir\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.538807 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.538771    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-tmp-dir\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.538839    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzks\" (UniqueName: \"kubernetes.io/projected/1fdda175-7281-4321-8554-d38310c31715-kube-api-access-xbzks\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.538865    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-model-cache\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.538904    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-home\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.538945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdda175-7281-4321-8554-d38310c31715-tls-certs\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.538968    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-dshm\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539296 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.539267    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-tmp-dir\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539296 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.539281    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-model-cache\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.539479 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.539460    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-home\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.541561 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.541529    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-dshm\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.541695 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.541680    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdda175-7281-4321-8554-d38310c31715-tls-certs\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.547136 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.547109    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzks\" (UniqueName: \"kubernetes.io/projected/1fdda175-7281-4321-8554-d38310c31715-kube-api-access-xbzks\") pod \"ipp-route-378627-kserve-7656759f9-fpq5z\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") " pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.625636 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.625547    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:36.779118 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:36.779096    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"]
Jul 24 18:28:36.781860 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:28:36.781829    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fdda175_7281_4321_8554_d38310c31715.slice/crio-531d422abdd5883239bdc4fbb8529750de0723387a1eca3c48830e03091666ac WatchSource:0}: Error finding container 531d422abdd5883239bdc4fbb8529750de0723387a1eca3c48830e03091666ac: Status 404 returned error can't find the container with id 531d422abdd5883239bdc4fbb8529750de0723387a1eca3c48830e03091666ac
Jul 24 18:28:37.632135 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:37.632102    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z" event={"ID":"1fdda175-7281-4321-8554-d38310c31715","Type":"ContainerStarted","Data":"d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571"}
Jul 24 18:28:37.632585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:37.632143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z" event={"ID":"1fdda175-7281-4321-8554-d38310c31715","Type":"ContainerStarted","Data":"531d422abdd5883239bdc4fbb8529750de0723387a1eca3c48830e03091666ac"}
Jul 24 18:28:37.650034 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:37.649983    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z" podStartSLOduration=1.6499699780000001 podStartE2EDuration="1.649969978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:28:37.648349769 +0000 UTC m=+2908.765277108" watchObservedRunningTime="2026-07-24 18:28:37.649969978 +0000 UTC m=+2908.766897318"
Jul 24 18:28:40.257257 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:40.257222    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:28:46.625848 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:46.625805    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:46.625848 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:46.625856    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:46.638289 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:46.638266    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:46.691046 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:46.691001    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:28:56.651069 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:28:56.651031    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:29:06.423050 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:29:06.423019    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-378627-746d9d945b-sh584_9b8f1899-cb68-4877-b4e0-1226638bcd78/payload-processing/0.log"
Jul 24 18:29:06.693937 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:29:06.693851    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b_454dbfc7-61bd-4371-b3c3-524d8d7b6a55/payload-processing/0.log"
Jul 24 18:29:23.549900 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:29:23.549858    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:29:47.757639 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:29:47.757559    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:30:09.494887 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:09.494860    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:30:09.503737 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:09.503715    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f2hht_37e312f0-8531-4f45-9267-2902b33b0788/ovn-acl-logging/0.log"
Jul 24 18:30:12.051017 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:12.050976    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:30:14.982175 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:14.982138    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"]
Jul 24 18:30:14.983172 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:14.982413    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584" podUID="9b8f1899-cb68-4877-b4e0-1226638bcd78" containerName="payload-processing" containerID="cri-o://9cc3b88a832187e085c17aa0a9187491a39afc7f8a2271d244543757e2010fe1" gracePeriod=30
Jul 24 18:30:14.987737 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:14.987705    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"]
Jul 24 18:30:14.988250 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:14.987933    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v" podUID="61308c9e-cae8-4e33-b17a-f7604213fad4" containerName="payload-pre-processing" containerID="cri-o://1a574dd04e1523d8ef3b4780c8d17b583121f5a880d7cf0d052f089bd1a6ccae" gracePeriod=30
Jul 24 18:30:15.156672 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.156638    2551 generic.go:356] "Generic (PLEG): container finished" podID="9b8f1899-cb68-4877-b4e0-1226638bcd78" containerID="9cc3b88a832187e085c17aa0a9187491a39afc7f8a2271d244543757e2010fe1" exitCode=0
Jul 24 18:30:15.156830 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.156714    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584" event={"ID":"9b8f1899-cb68-4877-b4e0-1226638bcd78","Type":"ContainerDied","Data":"9cc3b88a832187e085c17aa0a9187491a39afc7f8a2271d244543757e2010fe1"}
Jul 24 18:30:15.158522 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.158497    2551 generic.go:356] "Generic (PLEG): container finished" podID="61308c9e-cae8-4e33-b17a-f7604213fad4" containerID="1a574dd04e1523d8ef3b4780c8d17b583121f5a880d7cf0d052f089bd1a6ccae" exitCode=0
Jul 24 18:30:15.158677 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.158656    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v" event={"ID":"61308c9e-cae8-4e33-b17a-f7604213fad4","Type":"ContainerDied","Data":"1a574dd04e1523d8ef3b4780c8d17b583121f5a880d7cf0d052f089bd1a6ccae"}
Jul 24 18:30:15.272309 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.272287    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:30:15.275426 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.275404    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:30:15.340833 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.340803    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61308c9e-cae8-4e33-b17a-f7604213fad4-plugins-config-volume\") pod \"61308c9e-cae8-4e33-b17a-f7604213fad4\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") "
Jul 24 18:30:15.341006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.340852    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8f1899-cb68-4877-b4e0-1226638bcd78-plugins-config-volume\") pod \"9b8f1899-cb68-4877-b4e0-1226638bcd78\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") "
Jul 24 18:30:15.341006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.340894    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szph5\" (UniqueName: \"kubernetes.io/projected/61308c9e-cae8-4e33-b17a-f7604213fad4-kube-api-access-szph5\") pod \"61308c9e-cae8-4e33-b17a-f7604213fad4\" (UID: \"61308c9e-cae8-4e33-b17a-f7604213fad4\") "
Jul 24 18:30:15.341006 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.340953    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k88j5\" (UniqueName: \"kubernetes.io/projected/9b8f1899-cb68-4877-b4e0-1226638bcd78-kube-api-access-k88j5\") pod \"9b8f1899-cb68-4877-b4e0-1226638bcd78\" (UID: \"9b8f1899-cb68-4877-b4e0-1226638bcd78\") "
Jul 24 18:30:15.341268 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.341240    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8f1899-cb68-4877-b4e0-1226638bcd78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b8f1899-cb68-4877-b4e0-1226638bcd78" (UID: "9b8f1899-cb68-4877-b4e0-1226638bcd78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:30:15.341359 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.341240    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61308c9e-cae8-4e33-b17a-f7604213fad4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61308c9e-cae8-4e33-b17a-f7604213fad4" (UID: "61308c9e-cae8-4e33-b17a-f7604213fad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:30:15.343222 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.343189    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61308c9e-cae8-4e33-b17a-f7604213fad4-kube-api-access-szph5" (OuterVolumeSpecName: "kube-api-access-szph5") pod "61308c9e-cae8-4e33-b17a-f7604213fad4" (UID: "61308c9e-cae8-4e33-b17a-f7604213fad4"). InnerVolumeSpecName "kube-api-access-szph5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:30:15.343324 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.343276    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8f1899-cb68-4877-b4e0-1226638bcd78-kube-api-access-k88j5" (OuterVolumeSpecName: "kube-api-access-k88j5") pod "9b8f1899-cb68-4877-b4e0-1226638bcd78" (UID: "9b8f1899-cb68-4877-b4e0-1226638bcd78"). InnerVolumeSpecName "kube-api-access-k88j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:30:15.442331 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.442289    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8f1899-cb68-4877-b4e0-1226638bcd78-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:15.442331 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.442322    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szph5\" (UniqueName: \"kubernetes.io/projected/61308c9e-cae8-4e33-b17a-f7604213fad4-kube-api-access-szph5\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:15.442331 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.442333    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88j5\" (UniqueName: \"kubernetes.io/projected/9b8f1899-cb68-4877-b4e0-1226638bcd78-kube-api-access-k88j5\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:15.442600 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:15.442341    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61308c9e-cae8-4e33-b17a-f7604213fad4-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:16.165365 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.165331    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"
Jul 24 18:30:16.165836 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.165330    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584" event={"ID":"9b8f1899-cb68-4877-b4e0-1226638bcd78","Type":"ContainerDied","Data":"a1636903fd357220f94025f8283dd9ac7c6d44a59c6f28f3859882bb051c9e1c"}
Jul 24 18:30:16.165836 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.165474    2551 scope.go:117] "RemoveContainer" containerID="9cc3b88a832187e085c17aa0a9187491a39afc7f8a2271d244543757e2010fe1"
Jul 24 18:30:16.166945 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.166895    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"
Jul 24 18:30:16.166945 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.166933    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v" event={"ID":"61308c9e-cae8-4e33-b17a-f7604213fad4","Type":"ContainerDied","Data":"baa8e4c2dfff376765040e415aa88afbe2261831dfbc361b3ab724b3cd69a1f9"}
Jul 24 18:30:16.175283 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.175264    2551 scope.go:117] "RemoveContainer" containerID="1a574dd04e1523d8ef3b4780c8d17b583121f5a880d7cf0d052f089bd1a6ccae"
Jul 24 18:30:16.196019 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.195989    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"]
Jul 24 18:30:16.199540 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.199519    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-378627-746d9d945b-sh584"]
Jul 24 18:30:16.216529 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.216504    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"]
Jul 24 18:30:16.219769 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:16.219747    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-378627-79d68fc5b4-sf22v"]
Jul 24 18:30:17.301729 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:17.301695    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61308c9e-cae8-4e33-b17a-f7604213fad4" path="/var/lib/kubelet/pods/61308c9e-cae8-4e33-b17a-f7604213fad4/volumes"
Jul 24 18:30:17.302257 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:17.302237    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b8f1899-cb68-4877-b4e0-1226638bcd78" path="/var/lib/kubelet/pods/9b8f1899-cb68-4877-b4e0-1226638bcd78/volumes"
Jul 24 18:30:21.592227 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:21.592187    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"]
Jul 24 18:30:25.442251 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:25.442216    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:30:36.612717 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.612676    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z" podUID="1fdda175-7281-4321-8554-d38310c31715" containerName="main" containerID="cri-o://d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571" gracePeriod=45
Jul 24 18:30:36.678700 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.678656    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z" podUID="1fdda175-7281-4321-8554-d38310c31715" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused"
Jul 24 18:30:36.873590 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.873528    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:30:36.949463 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949431    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-model-cache\") pod \"1fdda175-7281-4321-8554-d38310c31715\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") "
Jul 24 18:30:36.949651 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949477    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-tmp-dir\") pod \"1fdda175-7281-4321-8554-d38310c31715\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") "
Jul 24 18:30:36.949651 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949502    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbzks\" (UniqueName: \"kubernetes.io/projected/1fdda175-7281-4321-8554-d38310c31715-kube-api-access-xbzks\") pod \"1fdda175-7281-4321-8554-d38310c31715\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") "
Jul 24 18:30:36.949651 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949526    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-dshm\") pod \"1fdda175-7281-4321-8554-d38310c31715\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") "
Jul 24 18:30:36.949651 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949613    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdda175-7281-4321-8554-d38310c31715-tls-certs\") pod \"1fdda175-7281-4321-8554-d38310c31715\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") "
Jul 24 18:30:36.949864 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949716    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-home\") pod \"1fdda175-7281-4321-8554-d38310c31715\" (UID: \"1fdda175-7281-4321-8554-d38310c31715\") "
Jul 24 18:30:36.949864 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949716    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fdda175-7281-4321-8554-d38310c31715" (UID: "1fdda175-7281-4321-8554-d38310c31715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:30:36.949864 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.949752    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fdda175-7281-4321-8554-d38310c31715" (UID: "1fdda175-7281-4321-8554-d38310c31715"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:30:36.950047 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.950021    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-home" (OuterVolumeSpecName: "home") pod "1fdda175-7281-4321-8554-d38310c31715" (UID: "1fdda175-7281-4321-8554-d38310c31715"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:30:36.950047 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.950040    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-model-cache\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:36.950133 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.950060    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-tmp-dir\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:36.951896 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.951871    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-dshm" (OuterVolumeSpecName: "dshm") pod "1fdda175-7281-4321-8554-d38310c31715" (UID: "1fdda175-7281-4321-8554-d38310c31715"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 18:30:36.951896 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.951883    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdda175-7281-4321-8554-d38310c31715-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fdda175-7281-4321-8554-d38310c31715" (UID: "1fdda175-7281-4321-8554-d38310c31715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:30:36.952007 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:36.951959    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdda175-7281-4321-8554-d38310c31715-kube-api-access-xbzks" (OuterVolumeSpecName: "kube-api-access-xbzks") pod "1fdda175-7281-4321-8554-d38310c31715" (UID: "1fdda175-7281-4321-8554-d38310c31715"). InnerVolumeSpecName "kube-api-access-xbzks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:30:37.051109 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.051075    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-home\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:37.051109 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.051102    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzks\" (UniqueName: \"kubernetes.io/projected/1fdda175-7281-4321-8554-d38310c31715-kube-api-access-xbzks\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:37.051109 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.051113    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdda175-7281-4321-8554-d38310c31715-dshm\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:37.051350 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.051122    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdda175-7281-4321-8554-d38310c31715-tls-certs\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:30:37.281329 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.281244    2551 generic.go:356] "Generic (PLEG): container finished" podID="1fdda175-7281-4321-8554-d38310c31715" containerID="d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571" exitCode=0
Jul 24 18:30:37.281329 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.281292    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"
Jul 24 18:30:37.281543 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.281333    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z" event={"ID":"1fdda175-7281-4321-8554-d38310c31715","Type":"ContainerDied","Data":"d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571"}
Jul 24 18:30:37.281543 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.281393    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z" event={"ID":"1fdda175-7281-4321-8554-d38310c31715","Type":"ContainerDied","Data":"531d422abdd5883239bdc4fbb8529750de0723387a1eca3c48830e03091666ac"}
Jul 24 18:30:37.281543 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.281417    2551 scope.go:117] "RemoveContainer" containerID="d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571"
Jul 24 18:30:37.290766 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.290749    2551 scope.go:117] "RemoveContainer" containerID="d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571"
Jul 24 18:30:37.291024 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:30:37.291004    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571\": container with ID starting with d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571 not found: ID does not exist" containerID="d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571"
Jul 24 18:30:37.291129 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.291033    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571"} err="failed to get container status \"d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571\": rpc error: code = NotFound desc = could not find container \"d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571\": container with ID starting with d4ac67d73cabeff0da8b8e087205b6616c34ecba0202be9af7336bf15afd0571 not found: ID does not exist"
Jul 24 18:30:37.309397 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.309356    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"]
Jul 24 18:30:37.311225 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:37.311200    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-378627/ipp-route-378627-kserve-7656759f9-fpq5z"]
Jul 24 18:30:39.300464 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:30:39.300431    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fdda175-7281-4321-8554-d38310c31715" path="/var/lib/kubelet/pods/1fdda175-7281-4321-8554-d38310c31715/volumes"
Jul 24 18:31:24.239809 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.239772    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"]
Jul 24 18:31:24.240348 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.240029    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b" podUID="454dbfc7-61bd-4371-b3c3-524d8d7b6a55" containerName="payload-processing" containerID="cri-o://e5c52f4a45dcc564c3f96d7086c26de909c8a76dab46ec478ffe099eb773b879" gracePeriod=30
Jul 24 18:31:24.246249 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.246215    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"]
Jul 24 18:31:24.246559 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.246528    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r" podUID="999889f7-5d6e-4398-b843-1cea4eeda865" containerName="payload-pre-processing" containerID="cri-o://1046afdf05342918fc09e68ad31940a9b351de08c0c1d74618bcc63545d54d18" gracePeriod=30
Jul 24 18:31:24.540744 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.540719    2551 generic.go:356] "Generic (PLEG): container finished" podID="454dbfc7-61bd-4371-b3c3-524d8d7b6a55" containerID="e5c52f4a45dcc564c3f96d7086c26de909c8a76dab46ec478ffe099eb773b879" exitCode=0
Jul 24 18:31:24.540859 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.540786    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b" event={"ID":"454dbfc7-61bd-4371-b3c3-524d8d7b6a55","Type":"ContainerDied","Data":"e5c52f4a45dcc564c3f96d7086c26de909c8a76dab46ec478ffe099eb773b879"}
Jul 24 18:31:24.542082 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.542060    2551 generic.go:356] "Generic (PLEG): container finished" podID="999889f7-5d6e-4398-b843-1cea4eeda865" containerID="1046afdf05342918fc09e68ad31940a9b351de08c0c1d74618bcc63545d54d18" exitCode=0
Jul 24 18:31:24.542183 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.542117    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r" event={"ID":"999889f7-5d6e-4398-b843-1cea4eeda865","Type":"ContainerDied","Data":"1046afdf05342918fc09e68ad31940a9b351de08c0c1d74618bcc63545d54d18"}
Jul 24 18:31:24.561032 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.561009    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:31:24.563787 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.563767    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:31:24.612950 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.612917    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-plugins-config-volume\") pod \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") "
Jul 24 18:31:24.613126 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.612999    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmtnl\" (UniqueName: \"kubernetes.io/projected/999889f7-5d6e-4398-b843-1cea4eeda865-kube-api-access-pmtnl\") pod \"999889f7-5d6e-4398-b843-1cea4eeda865\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") "
Jul 24 18:31:24.613126 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.613021    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999889f7-5d6e-4398-b843-1cea4eeda865-plugins-config-volume\") pod \"999889f7-5d6e-4398-b843-1cea4eeda865\" (UID: \"999889f7-5d6e-4398-b843-1cea4eeda865\") "
Jul 24 18:31:24.613126 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.613082    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8fh6\" (UniqueName: \"kubernetes.io/projected/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-kube-api-access-g8fh6\") pod \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\" (UID: \"454dbfc7-61bd-4371-b3c3-524d8d7b6a55\") "
Jul 24 18:31:24.613365 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.613335    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "454dbfc7-61bd-4371-b3c3-524d8d7b6a55" (UID: "454dbfc7-61bd-4371-b3c3-524d8d7b6a55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:24.613473 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.613409    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999889f7-5d6e-4398-b843-1cea4eeda865-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "999889f7-5d6e-4398-b843-1cea4eeda865" (UID: "999889f7-5d6e-4398-b843-1cea4eeda865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:24.615437 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.615411    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999889f7-5d6e-4398-b843-1cea4eeda865-kube-api-access-pmtnl" (OuterVolumeSpecName: "kube-api-access-pmtnl") pod "999889f7-5d6e-4398-b843-1cea4eeda865" (UID: "999889f7-5d6e-4398-b843-1cea4eeda865"). InnerVolumeSpecName "kube-api-access-pmtnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:24.615529 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.615409    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-kube-api-access-g8fh6" (OuterVolumeSpecName: "kube-api-access-g8fh6") pod "454dbfc7-61bd-4371-b3c3-524d8d7b6a55" (UID: "454dbfc7-61bd-4371-b3c3-524d8d7b6a55"). InnerVolumeSpecName "kube-api-access-g8fh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:24.714093 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.714053    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtnl\" (UniqueName: \"kubernetes.io/projected/999889f7-5d6e-4398-b843-1cea4eeda865-kube-api-access-pmtnl\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:24.714093 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.714085    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999889f7-5d6e-4398-b843-1cea4eeda865-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:24.714093 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.714099    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fh6\" (UniqueName: \"kubernetes.io/projected/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-kube-api-access-g8fh6\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:24.714350 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:24.714111    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454dbfc7-61bd-4371-b3c3-524d8d7b6a55-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:25.548049 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.548022    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"
Jul 24 18:31:25.548049 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.548035    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b" event={"ID":"454dbfc7-61bd-4371-b3c3-524d8d7b6a55","Type":"ContainerDied","Data":"4df596d462bd12008f8c72b62039d39f40844b758e96b8b07db74220123a56e9"}
Jul 24 18:31:25.548621 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.548106    2551 scope.go:117] "RemoveContainer" containerID="e5c52f4a45dcc564c3f96d7086c26de909c8a76dab46ec478ffe099eb773b879"
Jul 24 18:31:25.549553 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.549524    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"
Jul 24 18:31:25.549553 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.549529    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r" event={"ID":"999889f7-5d6e-4398-b843-1cea4eeda865","Type":"ContainerDied","Data":"7415c814dfe338807b2ee9fbf729fd62ab4b58fc33fe75be2b1da2a11b1091d5"}
Jul 24 18:31:25.557268 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.557248    2551 scope.go:117] "RemoveContainer" containerID="1046afdf05342918fc09e68ad31940a9b351de08c0c1d74618bcc63545d54d18"
Jul 24 18:31:25.566678 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.566659    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"]
Jul 24 18:31:25.569447 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.569426    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb47da-8584db98bd-kjw5r"]
Jul 24 18:31:25.581122 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.581094    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"]
Jul 24 18:31:25.583102 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:25.583083    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cb47da-6c84d7bd6-q787b"]
Jul 24 18:31:27.301083 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:27.301046    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454dbfc7-61bd-4371-b3c3-524d8d7b6a55" path="/var/lib/kubelet/pods/454dbfc7-61bd-4371-b3c3-524d8d7b6a55/volumes"
Jul 24 18:31:27.301553 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:27.301444    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="999889f7-5d6e-4398-b843-1cea4eeda865" path="/var/lib/kubelet/pods/999889f7-5d6e-4398-b843-1cea4eeda865/volumes"
Jul 24 18:31:27.748415 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:27.748312    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:31:46.620674 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.620639    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"]
Jul 24 18:31:46.621568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.621530    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b" podUID="5b85d54a-0332-40ff-9c45-3dec141c2258" containerName="maas-api" containerID="cri-o://c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb" gracePeriod=30
Jul 24 18:31:46.686780 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.686746    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"]
Jul 24 18:31:46.687084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.687025    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd" podUID="01051c77-e1e5-4938-8283-8746530c9971" containerName="payload-processing" containerID="cri-o://339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f" gracePeriod=30
Jul 24 18:31:46.692143 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.692113    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"]
Jul 24 18:31:46.692361 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.692341    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x" podUID="1544e5d0-2408-49bb-b583-971407d62377" containerName="payload-pre-processing" containerID="cri-o://90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419" gracePeriod=30
Jul 24 18:31:46.892588 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.892558    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:31:46.966704 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.966673    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:31:46.975135 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:46.975107    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:31:47.023956 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.023914    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85d54a-0332-40ff-9c45-3dec141c2258-openshift-service-ca\") pod \"5b85d54a-0332-40ff-9c45-3dec141c2258\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") "
Jul 24 18:31:47.024126 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.023968    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44p4j\" (UniqueName: \"kubernetes.io/projected/5b85d54a-0332-40ff-9c45-3dec141c2258-kube-api-access-44p4j\") pod \"5b85d54a-0332-40ff-9c45-3dec141c2258\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") "
Jul 24 18:31:47.024126 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.024030    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls\") pod \"5b85d54a-0332-40ff-9c45-3dec141c2258\" (UID: \"5b85d54a-0332-40ff-9c45-3dec141c2258\") "
Jul 24 18:31:47.024354 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.024329    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b85d54a-0332-40ff-9c45-3dec141c2258-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b85d54a-0332-40ff-9c45-3dec141c2258" (UID: "5b85d54a-0332-40ff-9c45-3dec141c2258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:47.026474 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.026441    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b85d54a-0332-40ff-9c45-3dec141c2258" (UID: "5b85d54a-0332-40ff-9c45-3dec141c2258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:31:47.026474 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.026451    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b85d54a-0332-40ff-9c45-3dec141c2258-kube-api-access-44p4j" (OuterVolumeSpecName: "kube-api-access-44p4j") pod "5b85d54a-0332-40ff-9c45-3dec141c2258" (UID: "5b85d54a-0332-40ff-9c45-3dec141c2258"). InnerVolumeSpecName "kube-api-access-44p4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:47.125560 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.125462    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldxkr\" (UniqueName: \"kubernetes.io/projected/01051c77-e1e5-4938-8283-8746530c9971-kube-api-access-ldxkr\") pod \"01051c77-e1e5-4938-8283-8746530c9971\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") "
Jul 24 18:31:47.125560 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.125506    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1544e5d0-2408-49bb-b583-971407d62377-plugins-config-volume\") pod \"1544e5d0-2408-49bb-b583-971407d62377\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") "
Jul 24 18:31:47.125801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.125590    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxpcq\" (UniqueName: \"kubernetes.io/projected/1544e5d0-2408-49bb-b583-971407d62377-kube-api-access-rxpcq\") pod \"1544e5d0-2408-49bb-b583-971407d62377\" (UID: \"1544e5d0-2408-49bb-b583-971407d62377\") "
Jul 24 18:31:47.125801 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.125657    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01051c77-e1e5-4938-8283-8746530c9971-plugins-config-volume\") pod \"01051c77-e1e5-4938-8283-8746530c9971\" (UID: \"01051c77-e1e5-4938-8283-8746530c9971\") "
Jul 24 18:31:47.126009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.125979    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b85d54a-0332-40ff-9c45-3dec141c2258-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:47.126009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.125988    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1544e5d0-2408-49bb-b583-971407d62377-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1544e5d0-2408-49bb-b583-971407d62377" (UID: "1544e5d0-2408-49bb-b583-971407d62377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:47.126009 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.126008    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44p4j\" (UniqueName: \"kubernetes.io/projected/5b85d54a-0332-40ff-9c45-3dec141c2258-kube-api-access-44p4j\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:47.126226 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.126022    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b85d54a-0332-40ff-9c45-3dec141c2258-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:47.126226 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.126053    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01051c77-e1e5-4938-8283-8746530c9971-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01051c77-e1e5-4938-8283-8746530c9971" (UID: "01051c77-e1e5-4938-8283-8746530c9971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:31:47.127812 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.127793    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01051c77-e1e5-4938-8283-8746530c9971-kube-api-access-ldxkr" (OuterVolumeSpecName: "kube-api-access-ldxkr") pod "01051c77-e1e5-4938-8283-8746530c9971" (UID: "01051c77-e1e5-4938-8283-8746530c9971"). InnerVolumeSpecName "kube-api-access-ldxkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:47.127884 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.127868    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1544e5d0-2408-49bb-b583-971407d62377-kube-api-access-rxpcq" (OuterVolumeSpecName: "kube-api-access-rxpcq") pod "1544e5d0-2408-49bb-b583-971407d62377" (UID: "1544e5d0-2408-49bb-b583-971407d62377"). InnerVolumeSpecName "kube-api-access-rxpcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:31:47.226600 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.226563    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01051c77-e1e5-4938-8283-8746530c9971-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:47.226600 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.226594    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxkr\" (UniqueName: \"kubernetes.io/projected/01051c77-e1e5-4938-8283-8746530c9971-kube-api-access-ldxkr\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:47.226600 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.226603    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1544e5d0-2408-49bb-b583-971407d62377-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:47.226842 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.226612    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpcq\" (UniqueName: \"kubernetes.io/projected/1544e5d0-2408-49bb-b583-971407d62377-kube-api-access-rxpcq\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:31:47.673903 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.673872    2551 generic.go:356] "Generic (PLEG): container finished" podID="01051c77-e1e5-4938-8283-8746530c9971" containerID="339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f" exitCode=0
Jul 24 18:31:47.674347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.673915    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"
Jul 24 18:31:47.674347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.673966    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd" event={"ID":"01051c77-e1e5-4938-8283-8746530c9971","Type":"ContainerDied","Data":"339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f"}
Jul 24 18:31:47.674347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.674018    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd" event={"ID":"01051c77-e1e5-4938-8283-8746530c9971","Type":"ContainerDied","Data":"ad4d93afcbea6d37583d4eda9ab1155293a81c0c72ac07203156e38671ec86a5"}
Jul 24 18:31:47.674347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.674047    2551 scope.go:117] "RemoveContainer" containerID="339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f"
Jul 24 18:31:47.675585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.675307    2551 generic.go:356] "Generic (PLEG): container finished" podID="1544e5d0-2408-49bb-b583-971407d62377" containerID="90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419" exitCode=0
Jul 24 18:31:47.675585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.675348    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"
Jul 24 18:31:47.675585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.675422    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x" event={"ID":"1544e5d0-2408-49bb-b583-971407d62377","Type":"ContainerDied","Data":"90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419"}
Jul 24 18:31:47.675585 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.675492    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x" event={"ID":"1544e5d0-2408-49bb-b583-971407d62377","Type":"ContainerDied","Data":"7fcc843d3dc61a67aa65f723a483b4427539e58f65e3c7d8370df757566839bd"}
Jul 24 18:31:47.677143 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.677096    2551 generic.go:356] "Generic (PLEG): container finished" podID="5b85d54a-0332-40ff-9c45-3dec141c2258" containerID="c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb" exitCode=0
Jul 24 18:31:47.677285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.677145    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b" event={"ID":"5b85d54a-0332-40ff-9c45-3dec141c2258","Type":"ContainerDied","Data":"c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb"}
Jul 24 18:31:47.677285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.677184    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b" event={"ID":"5b85d54a-0332-40ff-9c45-3dec141c2258","Type":"ContainerDied","Data":"1fc263f8bd97d41d0c814b90d383a3bbeaeec983c2a8a72146f7d3a9c332e35b"}
Jul 24 18:31:47.677285 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.677150    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"
Jul 24 18:31:47.684360 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.684339    2551 scope.go:117] "RemoveContainer" containerID="339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f"
Jul 24 18:31:47.684653 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:31:47.684631    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f\": container with ID starting with 339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f not found: ID does not exist" containerID="339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f"
Jul 24 18:31:47.684707 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.684662    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f"} err="failed to get container status \"339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f\": rpc error: code = NotFound desc = could not find container \"339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f\": container with ID starting with 339c9097796c886acdc1619b69069a98a4a033e198195d94a16d062eb05ff66f not found: ID does not exist"
Jul 24 18:31:47.684707 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.684680    2551 scope.go:117] "RemoveContainer" containerID="90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419"
Jul 24 18:31:47.693178 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.693161    2551 scope.go:117] "RemoveContainer" containerID="90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419"
Jul 24 18:31:47.693595 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:31:47.693545    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419\": container with ID starting with 90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419 not found: ID does not exist" containerID="90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419"
Jul 24 18:31:47.693595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.693579    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419"} err="failed to get container status \"90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419\": rpc error: code = NotFound desc = could not find container \"90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419\": container with ID starting with 90d2501586cd8083be02bdbd878700c78ce28bb118b348c6eca74d6098997419 not found: ID does not exist"
Jul 24 18:31:47.693595 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.693598    2551 scope.go:117] "RemoveContainer" containerID="c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb"
Jul 24 18:31:47.696608 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.696588    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"]
Jul 24 18:31:47.702397 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.702356    2551 scope.go:117] "RemoveContainer" containerID="c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb"
Jul 24 18:31:47.702785 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:31:47.702756    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb\": container with ID starting with c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb not found: ID does not exist" containerID="c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb"
Jul 24 18:31:47.702906 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.702795    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb"} err="failed to get container status \"c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb\": rpc error: code = NotFound desc = could not find container \"c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb\": container with ID starting with c7b2ba7c9e02d12c813f7cf1d03a16efca844eb7c197257207897becc6348aeb not found: ID does not exist"
Jul 24 18:31:47.705619 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.705600    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b90928-b699c8b96-jxs2b"]
Jul 24 18:31:47.715142 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.715121    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"]
Jul 24 18:31:47.719089 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.719066    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-b90928-7f6d55598b-jhcdd"]
Jul 24 18:31:47.729721 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.729700    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"]
Jul 24 18:31:47.733586 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:47.733565    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-b90928-f5d75f96-dt74x"]
Jul 24 18:31:49.301293 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:49.301257    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01051c77-e1e5-4938-8283-8746530c9971" path="/var/lib/kubelet/pods/01051c77-e1e5-4938-8283-8746530c9971/volumes"
Jul 24 18:31:49.301668 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:49.301638    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1544e5d0-2408-49bb-b583-971407d62377" path="/var/lib/kubelet/pods/1544e5d0-2408-49bb-b583-971407d62377/volumes"
Jul 24 18:31:49.301986 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:49.301973    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b85d54a-0332-40ff-9c45-3dec141c2258" path="/var/lib/kubelet/pods/5b85d54a-0332-40ff-9c45-3dec141c2258/volumes"
Jul 24 18:31:49.941369 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:31:49.941333    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:32:08.143568 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.143530    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"]
Jul 24 18:32:08.144342 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.144305    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4" podUID="f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" containerName="maas-api" containerID="cri-o://f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63" gracePeriod=30
Jul 24 18:32:08.209575 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.209535    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"]
Jul 24 18:32:08.209858 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.209819    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q" podUID="24da083a-e97e-4dea-ab12-a55f08463948" containerName="payload-processing" containerID="cri-o://06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09" gracePeriod=30
Jul 24 18:32:08.215161 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.215100    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"]
Jul 24 18:32:08.215395 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.215336    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm" podUID="f06bb3fc-d5af-41f0-9ebb-f71c6381f672" containerName="payload-pre-processing" containerID="cri-o://ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f" gracePeriod=30
Jul 24 18:32:08.305857 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.305815    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q" podUID="24da083a-e97e-4dea-ab12-a55f08463948" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.70:9004: connect: connection refused"
Jul 24 18:32:08.306044 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.305820    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm" podUID="f06bb3fc-d5af-41f0-9ebb-f71c6381f672" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.69:9004: connect: connection refused"
Jul 24 18:32:08.515090 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.515066    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:32:08.518111 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.518087    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:32:08.521725 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.521705    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:32:08.633169 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633130    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b4qk\" (UniqueName: \"kubernetes.io/projected/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-kube-api-access-2b4qk\") pod \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") "
Jul 24 18:32:08.633407 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633182    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-plugins-config-volume\") pod \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\" (UID: \"f06bb3fc-d5af-41f0-9ebb-f71c6381f672\") "
Jul 24 18:32:08.633407 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633217    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls\") pod \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") "
Jul 24 18:32:08.633407 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633273    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtmtn\" (UniqueName: \"kubernetes.io/projected/24da083a-e97e-4dea-ab12-a55f08463948-kube-api-access-gtmtn\") pod \"24da083a-e97e-4dea-ab12-a55f08463948\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") "
Jul 24 18:32:08.633407 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633301    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr92v\" (UniqueName: \"kubernetes.io/projected/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-kube-api-access-jr92v\") pod \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") "
Jul 24 18:32:08.633407 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633325    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24da083a-e97e-4dea-ab12-a55f08463948-plugins-config-volume\") pod \"24da083a-e97e-4dea-ab12-a55f08463948\" (UID: \"24da083a-e97e-4dea-ab12-a55f08463948\") "
Jul 24 18:32:08.633407 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633406    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-openshift-service-ca\") pod \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\" (UID: \"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6\") "
Jul 24 18:32:08.633734 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633640    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f06bb3fc-d5af-41f0-9ebb-f71c6381f672" (UID: "f06bb3fc-d5af-41f0-9ebb-f71c6381f672"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:32:08.633842 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633810    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:08.634196 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.633982    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" (UID: "f0547ac1-2ebf-4181-9a5d-29d1062fdfb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:32:08.634278 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.634120    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24da083a-e97e-4dea-ab12-a55f08463948-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24da083a-e97e-4dea-ab12-a55f08463948" (UID: "24da083a-e97e-4dea-ab12-a55f08463948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 18:32:08.635637 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.635612    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-kube-api-access-2b4qk" (OuterVolumeSpecName: "kube-api-access-2b4qk") pod "f06bb3fc-d5af-41f0-9ebb-f71c6381f672" (UID: "f06bb3fc-d5af-41f0-9ebb-f71c6381f672"). InnerVolumeSpecName "kube-api-access-2b4qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:32:08.635718 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.635683    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-kube-api-access-jr92v" (OuterVolumeSpecName: "kube-api-access-jr92v") pod "f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" (UID: "f0547ac1-2ebf-4181-9a5d-29d1062fdfb6"). InnerVolumeSpecName "kube-api-access-jr92v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:32:08.635951 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.635934    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da083a-e97e-4dea-ab12-a55f08463948-kube-api-access-gtmtn" (OuterVolumeSpecName: "kube-api-access-gtmtn") pod "24da083a-e97e-4dea-ab12-a55f08463948" (UID: "24da083a-e97e-4dea-ab12-a55f08463948"). InnerVolumeSpecName "kube-api-access-gtmtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:32:08.635995 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.635946    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" (UID: "f0547ac1-2ebf-4181-9a5d-29d1062fdfb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 18:32:08.735084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.735002    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmtn\" (UniqueName: \"kubernetes.io/projected/24da083a-e97e-4dea-ab12-a55f08463948-kube-api-access-gtmtn\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:08.735084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.735029    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr92v\" (UniqueName: \"kubernetes.io/projected/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-kube-api-access-jr92v\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:08.735084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.735039    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24da083a-e97e-4dea-ab12-a55f08463948-plugins-config-volume\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:08.735084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.735047    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-openshift-service-ca\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:08.735084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.735056    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b4qk\" (UniqueName: \"kubernetes.io/projected/f06bb3fc-d5af-41f0-9ebb-f71c6381f672-kube-api-access-2b4qk\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:08.735084 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.735066    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6-maas-api-tls\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:08.793382 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.793348    2551 generic.go:356] "Generic (PLEG): container finished" podID="f06bb3fc-d5af-41f0-9ebb-f71c6381f672" containerID="ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f" exitCode=0
Jul 24 18:32:08.793555 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.793405    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"
Jul 24 18:32:08.793555 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.793433    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm" event={"ID":"f06bb3fc-d5af-41f0-9ebb-f71c6381f672","Type":"ContainerDied","Data":"ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f"}
Jul 24 18:32:08.793555 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.793488    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm" event={"ID":"f06bb3fc-d5af-41f0-9ebb-f71c6381f672","Type":"ContainerDied","Data":"825afba92b835e60a9bc0624ca538f403d474677f314a0b47c014d0fc137ae95"}
Jul 24 18:32:08.793555 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.793515    2551 scope.go:117] "RemoveContainer" containerID="ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f"
Jul 24 18:32:08.794914 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.794883    2551 generic.go:356] "Generic (PLEG): container finished" podID="f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" containerID="f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63" exitCode=0
Jul 24 18:32:08.795008 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.794921    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"
Jul 24 18:32:08.795008 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.794921    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4" event={"ID":"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6","Type":"ContainerDied","Data":"f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63"}
Jul 24 18:32:08.795008 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.794965    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4" event={"ID":"f0547ac1-2ebf-4181-9a5d-29d1062fdfb6","Type":"ContainerDied","Data":"de0b19410c3610ebffaba9e8218974f60dc506e2a5e779418a238a34b754c490"}
Jul 24 18:32:08.796624 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.796598    2551 generic.go:356] "Generic (PLEG): container finished" podID="24da083a-e97e-4dea-ab12-a55f08463948" containerID="06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09" exitCode=0
Jul 24 18:32:08.796779 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.796632    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q" event={"ID":"24da083a-e97e-4dea-ab12-a55f08463948","Type":"ContainerDied","Data":"06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09"}
Jul 24 18:32:08.796911 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.796809    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q" event={"ID":"24da083a-e97e-4dea-ab12-a55f08463948","Type":"ContainerDied","Data":"286f17b559e0ce0c4ed4006d6176d70f4483c69c6acfb6ca593f9dfea37e49a2"}
Jul 24 18:32:08.796911 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.796651    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"
Jul 24 18:32:08.808119 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.808089    2551 scope.go:117] "RemoveContainer" containerID="ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f"
Jul 24 18:32:08.808487 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:32:08.808459    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f\": container with ID starting with ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f not found: ID does not exist" containerID="ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f"
Jul 24 18:32:08.808572 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.808498    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f"} err="failed to get container status \"ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f\": rpc error: code = NotFound desc = could not find container \"ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f\": container with ID starting with ef1a05dd25409a8c6b941be97024b79549630734cba0b14b142cad8ebfb9538f not found: ID does not exist"
Jul 24 18:32:08.808572 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.808521    2551 scope.go:117] "RemoveContainer" containerID="f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63"
Jul 24 18:32:08.816609 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.816592    2551 scope.go:117] "RemoveContainer" containerID="f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63"
Jul 24 18:32:08.816864 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:32:08.816845    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63\": container with ID starting with f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63 not found: ID does not exist" containerID="f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63"
Jul 24 18:32:08.816904 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.816872    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63"} err="failed to get container status \"f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63\": rpc error: code = NotFound desc = could not find container \"f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63\": container with ID starting with f1699fa95da990d746e7d326bd648dbf87d1ed08069cf553da923742cf150c63 not found: ID does not exist"
Jul 24 18:32:08.816904 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.816887    2551 scope.go:117] "RemoveContainer" containerID="06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09"
Jul 24 18:32:08.821931 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.821909    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"]
Jul 24 18:32:08.824987 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.824971    2551 scope.go:117] "RemoveContainer" containerID="06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09"
Jul 24 18:32:08.825236 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:32:08.825212    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09\": container with ID starting with 06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09 not found: ID does not exist" containerID="06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09"
Jul 24 18:32:08.825557 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.825351    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09"} err="failed to get container status \"06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09\": rpc error: code = NotFound desc = could not find container \"06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09\": container with ID starting with 06b71326a3dbbfda4c0773ce8c7a242778c988b95f97412610955382e29bca09 not found: ID does not exist"
Jul 24 18:32:08.827451 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.827434    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-88d4a7-67d797bfbd-sqmsm"]
Jul 24 18:32:08.837816 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.837796    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"]
Jul 24 18:32:08.841867 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.841847    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-88d4a7-7f459d748c-72tw4"]
Jul 24 18:32:08.852666 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.852646    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"]
Jul 24 18:32:08.856905 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:08.856884    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-88d4a7-76c5c6b88b-jkd2q"]
Jul 24 18:32:09.301602 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:09.301561    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24da083a-e97e-4dea-ab12-a55f08463948" path="/var/lib/kubelet/pods/24da083a-e97e-4dea-ab12-a55f08463948/volumes"
Jul 24 18:32:09.302153 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:09.302132    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" path="/var/lib/kubelet/pods/f0547ac1-2ebf-4181-9a5d-29d1062fdfb6/volumes"
Jul 24 18:32:09.302556 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:09.302537    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f06bb3fc-d5af-41f0-9ebb-f71c6381f672" path="/var/lib/kubelet/pods/f06bb3fc-d5af-41f0-9ebb-f71c6381f672/volumes"
Jul 24 18:32:11.150777 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:11.150739    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9gj8l"]
Jul 24 18:32:18.731095 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731056    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4tj6"]
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731623    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" containerName="maas-api"
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731643    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" containerName="maas-api"
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731659    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06bb3fc-d5af-41f0-9ebb-f71c6381f672" containerName="payload-pre-processing"
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731670    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06bb3fc-d5af-41f0-9ebb-f71c6381f672" containerName="payload-pre-processing"
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731681    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="999889f7-5d6e-4398-b843-1cea4eeda865" containerName="payload-pre-processing"
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731690    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="999889f7-5d6e-4398-b843-1cea4eeda865" containerName="payload-pre-processing"
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731701    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1544e5d0-2408-49bb-b583-971407d62377" containerName="payload-pre-processing"
Jul 24 18:32:18.731710 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731708    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1544e5d0-2408-49bb-b583-971407d62377" containerName="payload-pre-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731721    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b85d54a-0332-40ff-9c45-3dec141c2258" containerName="maas-api"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731729    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b85d54a-0332-40ff-9c45-3dec141c2258" containerName="maas-api"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731742    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01051c77-e1e5-4938-8283-8746530c9971" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731751    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="01051c77-e1e5-4938-8283-8746530c9971" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731768    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8f1899-cb68-4877-b4e0-1226638bcd78" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731776    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8f1899-cb68-4877-b4e0-1226638bcd78" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731789    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454dbfc7-61bd-4371-b3c3-524d8d7b6a55" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731797    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="454dbfc7-61bd-4371-b3c3-524d8d7b6a55" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731816    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24da083a-e97e-4dea-ab12-a55f08463948" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731825    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="24da083a-e97e-4dea-ab12-a55f08463948" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731839    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61308c9e-cae8-4e33-b17a-f7604213fad4" containerName="payload-pre-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731846    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="61308c9e-cae8-4e33-b17a-f7604213fad4" containerName="payload-pre-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731861    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdda175-7281-4321-8554-d38310c31715" containerName="main"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731869    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdda175-7281-4321-8554-d38310c31715" containerName="main"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731956    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="24da083a-e97e-4dea-ab12-a55f08463948" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731969    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="999889f7-5d6e-4398-b843-1cea4eeda865" containerName="payload-pre-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731978    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06bb3fc-d5af-41f0-9ebb-f71c6381f672" containerName="payload-pre-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731985    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1544e5d0-2408-49bb-b583-971407d62377" containerName="payload-pre-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.731992    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="61308c9e-cae8-4e33-b17a-f7604213fad4" containerName="payload-pre-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.732001    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b85d54a-0332-40ff-9c45-3dec141c2258" containerName="maas-api"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.732007    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fdda175-7281-4321-8554-d38310c31715" containerName="main"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.732019    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0547ac1-2ebf-4181-9a5d-29d1062fdfb6" containerName="maas-api"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.732028    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8f1899-cb68-4877-b4e0-1226638bcd78" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.732037    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="454dbfc7-61bd-4371-b3c3-524d8d7b6a55" containerName="payload-processing"
Jul 24 18:32:18.732147 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.732046    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="01051c77-e1e5-4938-8283-8746530c9971" containerName="payload-processing"
Jul 24 18:32:18.737291 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.737269    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:18.739874 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.739852    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-cltwp\""
Jul 24 18:32:18.745681 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.745660    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4tj6"]
Jul 24 18:32:18.825337 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.825292    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42tp8\" (UniqueName: \"kubernetes.io/projected/6112f04a-e2ce-44b6-9dec-00d1b0eebf6d-kube-api-access-42tp8\") pod \"kuadrant-operator-catalog-b4tj6\" (UID: \"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d\") " pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:18.926605 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.926568    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42tp8\" (UniqueName: \"kubernetes.io/projected/6112f04a-e2ce-44b6-9dec-00d1b0eebf6d-kube-api-access-42tp8\") pod \"kuadrant-operator-catalog-b4tj6\" (UID: \"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d\") " pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:18.936232 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:18.936202    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42tp8\" (UniqueName: \"kubernetes.io/projected/6112f04a-e2ce-44b6-9dec-00d1b0eebf6d-kube-api-access-42tp8\") pod \"kuadrant-operator-catalog-b4tj6\" (UID: \"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d\") " pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:19.047410 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:19.047298    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:19.193625 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:19.193322    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4tj6"]
Jul 24 18:32:19.195976 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:32:19.195943    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6112f04a_e2ce_44b6_9dec_00d1b0eebf6d.slice/crio-ac8a052026af112b37fa7a1790294d78f08e7d2ccf8a4e5ccdc744db467cfc58 WatchSource:0}: Error finding container ac8a052026af112b37fa7a1790294d78f08e7d2ccf8a4e5ccdc744db467cfc58: Status 404 returned error can't find the container with id ac8a052026af112b37fa7a1790294d78f08e7d2ccf8a4e5ccdc744db467cfc58
Jul 24 18:32:19.197480 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:19.197457    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 18:32:19.852391 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:19.852348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6" event={"ID":"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d","Type":"ContainerStarted","Data":"ac8a052026af112b37fa7a1790294d78f08e7d2ccf8a4e5ccdc744db467cfc58"}
Jul 24 18:32:21.863141 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:21.863106    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6" event={"ID":"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d","Type":"ContainerStarted","Data":"caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48"}
Jul 24 18:32:21.881696 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:21.881645    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6" podStartSLOduration=2.252998218 podStartE2EDuration="3.881629965s" totalImagesPullingTime="1.628631747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:32:21.880730069 +0000 UTC m=+3132.997657412" watchObservedRunningTime="2026-07-24 18:32:21.881629965 +0000 UTC m=+3132.998557306"
Jul 24 18:32:25.989235 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:25.989192    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d669b67d-2ckbr_89df1f4e-da55-454d-9573-245ef2699eec/manager/0.log"
Jul 24 18:32:26.628483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:26.628451    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f579866-wf9ck_d0e2302e-afe9-42e8-aa1b-c0b811b514b6/copy-manifests/0.log"
Jul 24 18:32:26.641031 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:26.641007    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f579866-wf9ck_d0e2302e-afe9-42e8-aa1b-c0b811b514b6/manager/0.log"
Jul 24 18:32:29.047929 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.047895    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:29.048348 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.047976    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:29.070497 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.070468    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:29.560690 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.560660    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-b4tj6_6112f04a-e2ce-44b6-9dec-00d1b0eebf6d/registry-server/0.log"
Jul 24 18:32:29.695753 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.695684    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vdmx5_2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9/manager/0.log"
Jul 24 18:32:29.816894 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.816815    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9gj8l_d4feebff-b7fe-4882-9e10-a5d80d993db4/limitador/0.log"
Jul 24 18:32:29.923840 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.923811    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:29.959044 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:29.959012    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4tj6"]
Jul 24 18:32:30.730392 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:30.730354    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddb4868f-vmnsz_a07eb4bd-43ef-4bcf-b546-5253e607e906/kube-auth-proxy/0.log"
Jul 24 18:32:31.910667 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:31.910627    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6" podUID="6112f04a-e2ce-44b6-9dec-00d1b0eebf6d" containerName="registry-server" containerID="cri-o://caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48" gracePeriod=2
Jul 24 18:32:32.177012 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.176987    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:32.252785 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.252755    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-rrl5l_b412203b-3299-4a66-b0da-fe81a7edbbdd/main/0.log"
Jul 24 18:32:32.355370 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.355336    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42tp8\" (UniqueName: \"kubernetes.io/projected/6112f04a-e2ce-44b6-9dec-00d1b0eebf6d-kube-api-access-42tp8\") pod \"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d\" (UID: \"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d\") "
Jul 24 18:32:32.357684 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.357648    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6112f04a-e2ce-44b6-9dec-00d1b0eebf6d-kube-api-access-42tp8" (OuterVolumeSpecName: "kube-api-access-42tp8") pod "6112f04a-e2ce-44b6-9dec-00d1b0eebf6d" (UID: "6112f04a-e2ce-44b6-9dec-00d1b0eebf6d"). InnerVolumeSpecName "kube-api-access-42tp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 18:32:32.456907 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.456821    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42tp8\" (UniqueName: \"kubernetes.io/projected/6112f04a-e2ce-44b6-9dec-00d1b0eebf6d-kube-api-access-42tp8\") on node \"ip-10-0-142-87.ec2.internal\" DevicePath \"\""
Jul 24 18:32:32.916865 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.916830    2551 generic.go:356] "Generic (PLEG): container finished" podID="6112f04a-e2ce-44b6-9dec-00d1b0eebf6d" containerID="caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48" exitCode=0
Jul 24 18:32:32.917294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.916874    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6"
Jul 24 18:32:32.917294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.916902    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6" event={"ID":"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d","Type":"ContainerDied","Data":"caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48"}
Jul 24 18:32:32.917294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.916944    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b4tj6" event={"ID":"6112f04a-e2ce-44b6-9dec-00d1b0eebf6d","Type":"ContainerDied","Data":"ac8a052026af112b37fa7a1790294d78f08e7d2ccf8a4e5ccdc744db467cfc58"}
Jul 24 18:32:32.917294 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.916962    2551 scope.go:117] "RemoveContainer" containerID="caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48"
Jul 24 18:32:32.926483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.926464    2551 scope.go:117] "RemoveContainer" containerID="caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48"
Jul 24 18:32:32.926765 ip-10-0-142-87 kubenswrapper[2551]: E0724 18:32:32.926743    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48\": container with ID starting with caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48 not found: ID does not exist" containerID="caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48"
Jul 24 18:32:32.926806 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.926774    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48"} err="failed to get container status \"caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48\": rpc error: code = NotFound desc = could not find container \"caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48\": container with ID starting with caa4c8eff5e8f5051b7a987ebf8fd2c9e62b5438220e7950d0a9054e5ea22f48 not found: ID does not exist"
Jul 24 18:32:32.937091 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.937067    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4tj6"]
Jul 24 18:32:32.940439 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:32.940417    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b4tj6"]
Jul 24 18:32:33.306919 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:33.306883    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6112f04a-e2ce-44b6-9dec-00d1b0eebf6d" path="/var/lib/kubelet/pods/6112f04a-e2ce-44b6-9dec-00d1b0eebf6d/volumes"
Jul 24 18:32:38.032351 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.032313    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhhl7/must-gather-5b995"]
Jul 24 18:32:38.032840 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.032728    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6112f04a-e2ce-44b6-9dec-00d1b0eebf6d" containerName="registry-server"
Jul 24 18:32:38.032840 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.032742    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6112f04a-e2ce-44b6-9dec-00d1b0eebf6d" containerName="registry-server"
Jul 24 18:32:38.032840 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.032828    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6112f04a-e2ce-44b6-9dec-00d1b0eebf6d" containerName="registry-server"
Jul 24 18:32:38.037453 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.037430    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.039771 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.039745    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhhl7\"/\"openshift-service-ca.crt\""
Jul 24 18:32:38.039916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.039895    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhhl7\"/\"default-dockercfg-hrkj5\""
Jul 24 18:32:38.040509 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.040495    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhhl7\"/\"kube-root-ca.crt\""
Jul 24 18:32:38.047483 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.047460    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhhl7/must-gather-5b995"]
Jul 24 18:32:38.104176 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.104135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee606c5c-aed5-4b8d-b351-3f51b56b6950-must-gather-output\") pod \"must-gather-5b995\" (UID: \"ee606c5c-aed5-4b8d-b351-3f51b56b6950\") " pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.104346 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.104214    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdc2\" (UniqueName: \"kubernetes.io/projected/ee606c5c-aed5-4b8d-b351-3f51b56b6950-kube-api-access-8mdc2\") pod \"must-gather-5b995\" (UID: \"ee606c5c-aed5-4b8d-b351-3f51b56b6950\") " pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.205436 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.205367    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee606c5c-aed5-4b8d-b351-3f51b56b6950-must-gather-output\") pod \"must-gather-5b995\" (UID: \"ee606c5c-aed5-4b8d-b351-3f51b56b6950\") " pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.205582 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.205483    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdc2\" (UniqueName: \"kubernetes.io/projected/ee606c5c-aed5-4b8d-b351-3f51b56b6950-kube-api-access-8mdc2\") pod \"must-gather-5b995\" (UID: \"ee606c5c-aed5-4b8d-b351-3f51b56b6950\") " pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.205725 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.205705    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee606c5c-aed5-4b8d-b351-3f51b56b6950-must-gather-output\") pod \"must-gather-5b995\" (UID: \"ee606c5c-aed5-4b8d-b351-3f51b56b6950\") " pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.214916 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.214889    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdc2\" (UniqueName: \"kubernetes.io/projected/ee606c5c-aed5-4b8d-b351-3f51b56b6950-kube-api-access-8mdc2\") pod \"must-gather-5b995\" (UID: \"ee606c5c-aed5-4b8d-b351-3f51b56b6950\") " pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.348244 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.348205    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhhl7/must-gather-5b995"
Jul 24 18:32:38.489984 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.489952    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhhl7/must-gather-5b995"]
Jul 24 18:32:38.493536 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:32:38.493510    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee606c5c_aed5_4b8d_b351_3f51b56b6950.slice/crio-b8f73ef52642d2ab623e2bc39b0b49041b647138e0916867e2f0e3494b3fee0a WatchSource:0}: Error finding container b8f73ef52642d2ab623e2bc39b0b49041b647138e0916867e2f0e3494b3fee0a: Status 404 returned error can't find the container with id b8f73ef52642d2ab623e2bc39b0b49041b647138e0916867e2f0e3494b3fee0a
Jul 24 18:32:38.947397 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:38.947343    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhhl7/must-gather-5b995" event={"ID":"ee606c5c-aed5-4b8d-b351-3f51b56b6950","Type":"ContainerStarted","Data":"b8f73ef52642d2ab623e2bc39b0b49041b647138e0916867e2f0e3494b3fee0a"}
Jul 24 18:32:39.969903 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:39.968596    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhhl7/must-gather-5b995" event={"ID":"ee606c5c-aed5-4b8d-b351-3f51b56b6950","Type":"ContainerStarted","Data":"eb99fe9b6788487013045940a90216193dcbb67f5eb36ab1e31176f869d1b0d1"}
Jul 24 18:32:39.969903 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:39.968672    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhhl7/must-gather-5b995" event={"ID":"ee606c5c-aed5-4b8d-b351-3f51b56b6950","Type":"ContainerStarted","Data":"a9f002fc7ff6e816a9674e8b2760b632cdea829435b69f976f2642f291cecf48"}
Jul 24 18:32:39.989197 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:39.989132    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhhl7/must-gather-5b995" podStartSLOduration=1.2473928459999999 podStartE2EDuration="1.989111463s" totalImagesPullingTime="741.718617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:32:39.98903061 +0000 UTC m=+3151.105957966" watchObservedRunningTime="2026-07-24 18:32:39.989111463 +0000 UTC m=+3151.106038803"
Jul 24 18:32:40.864738 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:40.864697    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f5gq5_d346c91f-4715-48f3-873f-a9cb78c34f74/global-pull-secret-syncer/0.log"
Jul 24 18:32:41.022931 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:41.022902    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q2fwc_7262267a-f238-42c0-944c-ba2f53479a7b/konnectivity-agent/0.log"
Jul 24 18:32:41.095803 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:41.095768    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-87.ec2.internal_6a6f6531a3f15fe4c59b4f31695fab0d/haproxy/0.log"
Jul 24 18:32:46.408201 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:46.408156    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vdmx5_2d49a6cd-cdea-44d4-abb0-d58bb9ef8da9/manager/0.log"
Jul 24 18:32:46.434105 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:46.434071    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9gj8l_d4feebff-b7fe-4882-9e10-a5d80d993db4/limitador/0.log"
Jul 24 18:32:48.374397 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.373989    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64893af6-084d-431a-b0d2-846b467493f4/alertmanager/0.log"
Jul 24 18:32:48.404102 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.404062    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64893af6-084d-431a-b0d2-846b467493f4/config-reloader/0.log"
Jul 24 18:32:48.428160 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.427939    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64893af6-084d-431a-b0d2-846b467493f4/kube-rbac-proxy-web/0.log"
Jul 24 18:32:48.460429 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.460404    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64893af6-084d-431a-b0d2-846b467493f4/kube-rbac-proxy/0.log"
Jul 24 18:32:48.484252 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.484226    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64893af6-084d-431a-b0d2-846b467493f4/kube-rbac-proxy-metric/0.log"
Jul 24 18:32:48.513219 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.513186    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64893af6-084d-431a-b0d2-846b467493f4/prom-label-proxy/0.log"
Jul 24 18:32:48.537724 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.537698    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_64893af6-084d-431a-b0d2-846b467493f4/init-config-reloader/0.log"
Jul 24 18:32:48.618270 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.618240    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-nfmll_73df4cb3-0d16-488a-a859-4d1a5de021d2/kube-state-metrics/0.log"
Jul 24 18:32:48.639231 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.639205    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-nfmll_73df4cb3-0d16-488a-a859-4d1a5de021d2/kube-rbac-proxy-main/0.log"
Jul 24 18:32:48.675173 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.675132    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-nfmll_73df4cb3-0d16-488a-a859-4d1a5de021d2/kube-rbac-proxy-self/0.log"
Jul 24 18:32:48.705080 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.704964    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86764cc475-q6smq_5de00dc6-3126-46a4-969d-dff272cfaff2/metrics-server/0.log"
Jul 24 18:32:48.735491 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.734907    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-v67b8_24b312cc-b411-4e2d-ac8a-af2c28b0c83e/monitoring-plugin/0.log"
Jul 24 18:32:48.770013 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.769985    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4bwj9_c2bbb19c-ed35-4308-a300-75b0e4021dd8/node-exporter/0.log"
Jul 24 18:32:48.791910 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.791880    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4bwj9_c2bbb19c-ed35-4308-a300-75b0e4021dd8/kube-rbac-proxy/0.log"
Jul 24 18:32:48.813403 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:48.813354    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4bwj9_c2bbb19c-ed35-4308-a300-75b0e4021dd8/init-textfile/0.log"
Jul 24 18:32:49.008152 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.008121    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pxsnj_f02594fb-33a4-432f-84c4-1937c3077ccb/kube-rbac-proxy-main/0.log"
Jul 24 18:32:49.034897 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.034860    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pxsnj_f02594fb-33a4-432f-84c4-1937c3077ccb/kube-rbac-proxy-self/0.log"
Jul 24 18:32:49.062115 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.062063    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pxsnj_f02594fb-33a4-432f-84c4-1937c3077ccb/openshift-state-metrics/0.log"
Jul 24 18:32:49.107681 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.107647    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36fc9504-4e7f-4890-932a-e335c670cfeb/prometheus/0.log"
Jul 24 18:32:49.136649 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.136612    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36fc9504-4e7f-4890-932a-e335c670cfeb/config-reloader/0.log"
Jul 24 18:32:49.166856 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.166825    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36fc9504-4e7f-4890-932a-e335c670cfeb/thanos-sidecar/0.log"
Jul 24 18:32:49.193655 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.193615    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36fc9504-4e7f-4890-932a-e335c670cfeb/kube-rbac-proxy-web/0.log"
Jul 24 18:32:49.219438 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.219339    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36fc9504-4e7f-4890-932a-e335c670cfeb/kube-rbac-proxy/0.log"
Jul 24 18:32:49.247715 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.247599    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36fc9504-4e7f-4890-932a-e335c670cfeb/kube-rbac-proxy-thanos/0.log"
Jul 24 18:32:49.270404 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.270357    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_36fc9504-4e7f-4890-932a-e335c670cfeb/init-config-reloader/0.log"
Jul 24 18:32:49.307588 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.307546    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-fph7j_e289892c-3e93-4479-a726-c9ff4896b0d2/prometheus-operator/0.log"
Jul 24 18:32:49.345182 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.345142    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-fph7j_e289892c-3e93-4479-a726-c9ff4896b0d2/kube-rbac-proxy/0.log"
Jul 24 18:32:49.410556 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.410524    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcc569685-tk72l_4403fc70-3600-4f5f-bd5d-3320f9fa9f7a/telemeter-client/0.log"
Jul 24 18:32:49.434320 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.434292    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcc569685-tk72l_4403fc70-3600-4f5f-bd5d-3320f9fa9f7a/reload/0.log"
Jul 24 18:32:49.460261 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.460198    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcc569685-tk72l_4403fc70-3600-4f5f-bd5d-3320f9fa9f7a/kube-rbac-proxy/0.log"
Jul 24 18:32:49.493534 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.493454    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74b54598bc-s4nqw_3adea86c-fe57-4909-b53c-b48e12f69fc6/thanos-query/0.log"
Jul 24 18:32:49.531363 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.531328    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74b54598bc-s4nqw_3adea86c-fe57-4909-b53c-b48e12f69fc6/kube-rbac-proxy-web/0.log"
Jul 24 18:32:49.555832 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.555804    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74b54598bc-s4nqw_3adea86c-fe57-4909-b53c-b48e12f69fc6/kube-rbac-proxy/0.log"
Jul 24 18:32:49.579523 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.579496    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74b54598bc-s4nqw_3adea86c-fe57-4909-b53c-b48e12f69fc6/prom-label-proxy/0.log"
Jul 24 18:32:49.602482 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.602447    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74b54598bc-s4nqw_3adea86c-fe57-4909-b53c-b48e12f69fc6/kube-rbac-proxy-rules/0.log"
Jul 24 18:32:49.625401 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:49.625347    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74b54598bc-s4nqw_3adea86c-fe57-4909-b53c-b48e12f69fc6/kube-rbac-proxy-metrics/0.log"
Jul 24 18:32:51.213071 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.213026    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"]
Jul 24 18:32:51.222215 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.222178    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.226055 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.226020    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"]
Jul 24 18:32:51.350151 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.350111    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-podres\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.350347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.350165    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-sys\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.350347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.350196    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplqn\" (UniqueName: \"kubernetes.io/projected/cb6f0931-cc89-4f8e-9a03-e221d589ee72-kube-api-access-xplqn\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.350347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.350253    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-proc\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.350347 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.350311    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-lib-modules\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451427 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451395    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-lib-modules\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451494    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-podres\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451517    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-sys\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451533    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplqn\" (UniqueName: \"kubernetes.io/projected/cb6f0931-cc89-4f8e-9a03-e221d589ee72-kube-api-access-xplqn\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451598 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451579    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-proc\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451738 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451614    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-lib-modules\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451738 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451668    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-proc\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451738 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451667    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-podres\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.451738 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.451692    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb6f0931-cc89-4f8e-9a03-e221d589ee72-sys\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.461434 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.461399    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplqn\" (UniqueName: \"kubernetes.io/projected/cb6f0931-cc89-4f8e-9a03-e221d589ee72-kube-api-access-xplqn\") pod \"perf-node-gather-daemonset-xv7kz\" (UID: \"cb6f0931-cc89-4f8e-9a03-e221d589ee72\") " pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.540616 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.540527    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:51.735643 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:51.735608    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"]
Jul 24 18:32:51.741182 ip-10-0-142-87 kubenswrapper[2551]: W0724 18:32:51.741131    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb6f0931_cc89_4f8e_9a03_e221d589ee72.slice/crio-68f0305c0e6cae2ff18d0a395256a44d62b9de3b1f3d8e079898df3d3907ddb7 WatchSource:0}: Error finding container 68f0305c0e6cae2ff18d0a395256a44d62b9de3b1f3d8e079898df3d3907ddb7: Status 404 returned error can't find the container with id 68f0305c0e6cae2ff18d0a395256a44d62b9de3b1f3d8e079898df3d3907ddb7
Jul 24 18:32:52.073089 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:52.072989    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz" event={"ID":"cb6f0931-cc89-4f8e-9a03-e221d589ee72","Type":"ContainerStarted","Data":"c6499c727f852e3ef4e69a86cfff6f74800f7cc6a06ba96805070b27fb584a5c"}
Jul 24 18:32:52.073089 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:52.073048    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz" event={"ID":"cb6f0931-cc89-4f8e-9a03-e221d589ee72","Type":"ContainerStarted","Data":"68f0305c0e6cae2ff18d0a395256a44d62b9de3b1f3d8e079898df3d3907ddb7"}
Jul 24 18:32:52.073335 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:52.073124    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:52.091709 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:52.091645    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz" podStartSLOduration=1.09162459 podStartE2EDuration="1.09162459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:32:52.08848868 +0000 UTC m=+3163.205416020" watchObservedRunningTime="2026-07-24 18:32:52.09162459 +0000 UTC m=+3163.208551932"
Jul 24 18:32:53.696110 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:53.696082    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c67v_f2879b37-cd85-4cbf-9899-657d371e03b0/dns/0.log"
Jul 24 18:32:53.717859 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:53.717836    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5c67v_f2879b37-cd85-4cbf-9899-657d371e03b0/kube-rbac-proxy/0.log"
Jul 24 18:32:53.836174 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:53.836141    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84cpg_1b44312a-d41c-4158-addc-4fd4b7a0c61f/dns-node-resolver/0.log"
Jul 24 18:32:54.462889 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:54.462859    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5x8nc_38c17424-6bce-4314-a59d-e0034753af29/node-ca/0.log"
Jul 24 18:32:55.622852 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:55.622820    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddb4868f-vmnsz_a07eb4bd-43ef-4bcf-b546-5253e607e906/kube-auth-proxy/0.log"
Jul 24 18:32:56.590067 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:56.590032    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kfr4f_361a1a91-150a-472a-9950-49eafdb49c3b/serve-healthcheck-canary/0.log"
Jul 24 18:32:57.153496 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:57.153466    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rc7j7_00eca871-94cf-449c-aac1-ec4971d7a909/kube-rbac-proxy/0.log"
Jul 24 18:32:57.175643 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:57.175614    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rc7j7_00eca871-94cf-449c-aac1-ec4971d7a909/exporter/0.log"
Jul 24 18:32:57.198826 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:57.198797    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rc7j7_00eca871-94cf-449c-aac1-ec4971d7a909/extractor/0.log"
Jul 24 18:32:58.090302 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:58.090277    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhhl7/perf-node-gather-daemonset-xv7kz"
Jul 24 18:32:59.917007 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:32:59.916974    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84d669b67d-2ckbr_89df1f4e-da55-454d-9573-245ef2699eec/manager/0.log"
Jul 24 18:33:00.246855 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:00.246819    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f579866-wf9ck_d0e2302e-afe9-42e8-aa1b-c0b811b514b6/manager/0.log"
Jul 24 18:33:00.265731 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:00.265693    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f579866-wf9ck_d0e2302e-afe9-42e8-aa1b-c0b811b514b6/copy-manifests/0.log"
Jul 24 18:33:01.672221 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:01.672190    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74c56c9c54-dkzpr_68acbd59-c8d8-483e-a933-68e1158f73f7/manager/0.log"
Jul 24 18:33:01.753173 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:01.753059    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddb4868f-vmnsz_a07eb4bd-43ef-4bcf-b546-5253e607e906/kube-auth-proxy/0.log"
Jul 24 18:33:03.419338 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:03.419267    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddb4868f-vmnsz_a07eb4bd-43ef-4bcf-b546-5253e607e906/kube-auth-proxy/0.log"
Jul 24 18:33:08.112877 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.112852    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bnscl_2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521/kube-multus-additional-cni-plugins/0.log"
Jul 24 18:33:08.135582 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.135561    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bnscl_2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521/egress-router-binary-copy/0.log"
Jul 24 18:33:08.157236 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.157220    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bnscl_2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521/cni-plugins/0.log"
Jul 24 18:33:08.178438 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.178413    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bnscl_2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521/bond-cni-plugin/0.log"
Jul 24 18:33:08.199536 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.199515    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bnscl_2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521/routeoverride-cni/0.log"
Jul 24 18:33:08.220108 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.220078    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bnscl_2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521/whereabouts-cni-bincopy/0.log"
Jul 24 18:33:08.241173 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.241147    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bnscl_2dc35ddf-9a6d-41cb-8b4d-92cf7a83c521/whereabouts-cni/0.log"
Jul 24 18:33:08.610848 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.610806    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n5qts_f63fe1ed-3d2a-41bb-837c-77b6beb9cb54/kube-multus/0.log"
Jul 24 18:33:08.774838 ip-10-0-142-87 kubenswrapper[2551]: I0724 18:33:08.774810    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wnsct_ca6bd76d-63fe-4c44-a496-6f9ab029b2cb/network-metrics-daemon/0.log"