Sep 10 17:11:18.203430 ip-10-0-142-52 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 17:11:18.203437 ip-10-0-142-52 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 17:11:18.203447 ip-10-0-142-52 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 17:11:18.203655 ip-10-0-142-52 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 17:11:28.351140 ip-10-0-142-52 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 17:11:28.351158 ip-10-0-142-52 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5cae5ab9a62e46ee946f843a3fb6d427 --
Sep 10 17:13:24.530824 ip-10-0-142-52 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 17:13:24.889824 ip-10-0-142-52 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:13:24.889824 ip-10-0-142-52 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 17:13:24.889824 ip-10-0-142-52 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:13:24.889824 ip-10-0-142-52 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 17:13:24.889824 ip-10-0-142-52 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:13:24.892210 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.892121    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 17:13:24.897578 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897562    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:13:24.897578 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897577    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897581    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897585    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897588    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897591    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897594    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897597    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897601    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897604    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897607    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897615    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897618    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897621    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897624    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897627    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897630    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897632    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897635    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897637    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897640    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:13:24.897654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897643    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897647    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897651    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897654    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897657    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897660    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897664    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897668    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897672    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897675    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897678    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897681    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897684    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897687    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897690    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897692    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897695    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897698    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897700    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897703    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:13:24.898119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897706    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897709    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897711    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897714    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897716    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897719    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897721    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897724    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897726    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897729    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897731    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897734    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897736    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897739    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897743    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897745    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897748    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897751    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897754    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897756    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:13:24.898631 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897759    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897761    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897764    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897767    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897769    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897772    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897774    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897777    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897779    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897782    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897785    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897788    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897790    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897793    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897795    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897798    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897800    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897803    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897805    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897808    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:13:24.899135 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897810    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897814    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897816    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897819    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897822    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897824    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897829    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897832    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897835    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897838    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897842    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897845    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897848    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897851    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897855    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897858    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897861    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.897863    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898624    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:13:24.899670 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898631    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898635    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898639    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898643    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898647    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898650    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898653    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898655    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898658    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898660    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898663    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898666    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898668    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898671    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898674    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898677    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898679    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898682    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898685    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898688    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:13:24.900136 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898690    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898693    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898695    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898698    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898701    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898703    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898706    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898709    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898712    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898714    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898717    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898719    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898722    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898724    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898727    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898729    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898732    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898735    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898738    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:13:24.900650 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898740    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898743    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898746    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898749    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898753    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898757    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898760    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898763    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898765    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898768    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898771    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898773    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898776    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898779    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898781    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898784    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898787    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898789    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898792    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898794    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:13:24.901119 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898798    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898800    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898802    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898805    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898808    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898810    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898812    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898815    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898817    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898820    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898822    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898825    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898828    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898830    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898833    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898836    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898838    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898841    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898843    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898846    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:13:24.901635 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898849    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898851    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898854    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898856    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898859    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898861    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898864    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898867    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898869    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898872    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898874    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898877    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898879    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898882    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898884    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898887    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898889    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898893    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.898897    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899288    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:13:24.902122 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899294    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899297    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899300    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899304    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899307    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899310    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899312    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899315    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899320    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899323    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899326    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899329    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899332    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899335    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899338    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899340    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899343    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899346    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899348    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899351    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:13:24.902630 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899353    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899356    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899359    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899362    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899364    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899367    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899370    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899373    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899375    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899378    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899383    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899385    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899388    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899390    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899393    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899395    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899398    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899402    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899405    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:13:24.903127 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899407    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899410    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899412    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899415    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899417    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899420    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899423    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899425    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899428    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899431    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899433    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899436    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899438    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899441    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899444    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899446    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899448    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899451    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899453    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899456    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:13:24.903624 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899459    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899462    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899465    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899469    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899472    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899474    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899477    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899480    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899482    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899485    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899488    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899490    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899493    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899495    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899498    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899500    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899503    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899505    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899508    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899510    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:13:24.904114 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899513    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899515    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899518    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899520    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899522    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899538    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899541    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899544    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899546    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899549    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899552    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899555    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899559    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899561    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899564    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899568    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899571    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899574    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.899577    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899652    2554 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 17:13:24.904653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899704    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899711    2554 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899715    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899721    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899729    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899733    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899737    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899741    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899744    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899747    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899750    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899754    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899756    2554 flags.go:64] FLAG: --cgroup-root=""
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899759    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899762    2554 flags.go:64] FLAG: --client-ca-file=""
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899765    2554 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899768    2554 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899773    2554 flags.go:64] FLAG: --cluster-domain=""
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899775    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899779    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899782    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899785    2554 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899790    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 17:13:24.905126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899793    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899796    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899799    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899802    2554 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899806    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899811    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899814    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899817    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899821    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899824    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899827    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899830    2554 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899833    2554 flags.go:64] FLAG: --enable-server="true"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899836    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899840    2554 flags.go:64] FLAG: --event-burst="100"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899843    2554 flags.go:64] FLAG: --event-qps="50"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899847    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899850    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899853    2554 flags.go:64] FLAG: --eviction-hard=""
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899856    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899859    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899862    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899865    2554 flags.go:64] FLAG: --eviction-soft=""
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899868    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899870    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 17:13:24.905701 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899873    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899876    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899879    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899882    2554 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899885    2554 flags.go:64] FLAG: --feature-gates=""
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899889    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899892    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899895    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899898    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899901    2554 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899904    2554 flags.go:64] FLAG: --help="false"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899907    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-142-52.ec2.internal"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899912    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899915    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899918    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899921    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899925    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899928    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899931    2554 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899934    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899937    2554 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899941    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899944    2554 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899947    2554 flags.go:64] FLAG: --kube-reserved=""
Sep 10 17:13:24.906315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899950    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899953    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899955    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899958    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899961    2554 flags.go:64] FLAG: --lock-file=""
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899964    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899967    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899970    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899975    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899978    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899981    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899983    2554 flags.go:64] FLAG: --logging-format="text"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899987    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899990    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899993    2554 flags.go:64] FLAG: --manifest-url=""
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.899995    2554 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900000    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900003    2554 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900007    2554 flags.go:64] FLAG: --max-pods="110"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900010    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900015    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900018    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900021    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900024    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900027    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 17:13:24.906934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900030    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900037    2554 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900040    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900043    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900046    2554 flags.go:64] FLAG: --pod-cidr=""
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900050    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900054    2554 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900057    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900060    2554 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900063    2554 flags.go:64] FLAG: --port="10250"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900066    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900069    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08f7f1d0a753e5d2d"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900072    2554 flags.go:64] FLAG: --qos-reserved=""
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900075    2554 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900078    2554 flags.go:64] FLAG: --register-node="true"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900081    2554 flags.go:64] FLAG: --register-with-taints=""
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900084    2554 flags.go:64] FLAG: --registry-burst="10"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900087    2554 flags.go:64] FLAG: --registry-qps="5"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900090    2554 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900093    2554 flags.go:64] FLAG: --reserved-memory=""
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900096    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900099    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900102    2554 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900105    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900108    2554 flags.go:64] FLAG: --runonce="false"
Sep 10 17:13:24.907607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900111    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900114    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900116    2554 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900121    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900124    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900127    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900130    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900133    2554 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900135    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900138    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900141    2554 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900144    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900148    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900151    2554 flags.go:64] FLAG: --system-cgroups=""
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900154    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900160    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900162    2554 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900166    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900170    2554 flags.go:64] FLAG: --tls-min-version=""
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900173    2554 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900175    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900178    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900181    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900184    2554 flags.go:64] FLAG: --v="2"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900188    2554 flags.go:64] FLAG: --version="false"
Sep 10 17:13:24.908250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900193    2554 flags.go:64] FLAG: --vmodule=""
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900197    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.900200    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900317    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900320    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900324    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900326    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900329    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900332    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900334    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900339    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900341    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900344    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900347    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900349    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900352    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900354    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900357    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900361    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:13:24.908883 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900365    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900368    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900371    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900374    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900376    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900379    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900382    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900384    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900387    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900390    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900393    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900397    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900400    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900404    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900407    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900411    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900413    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900416    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900419    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:13:24.909347 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900422    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900425    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900427    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900430    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900434    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900436    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900439    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900442    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900444    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900447    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900449    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900452    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900455    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900459    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900463    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900467    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900470    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900473    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900476    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900478    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:13:24.909827 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900481    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900484    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900487    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900489    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900492    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900494    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900498    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900501    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900504    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900506    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900509    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900513    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900516    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900518    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900521    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900524    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900541    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900544    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900547    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:13:24.910306 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900550    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900553    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900555    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900558    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900560    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900563    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900565    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900568    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900571    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900573    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900577    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900580    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900583    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900585    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900588    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900591    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900593    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900596    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900598    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900603    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:13:24.910785 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900605    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900608    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900610    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900613    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.900615    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.901185    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.908514    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.908545    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.909300    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.909307    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909383    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909388    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909392    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909395    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909398    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:13:24.911279 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909402    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909405    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909408    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909411    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909413    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909415    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909418    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909420    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909423    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909426    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909428    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909431    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909434    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909436    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909439    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909441    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909444    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909446    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909449    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909451    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:13:24.911707 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909454    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909456    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909458    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909461    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909465    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909468    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909470    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909473    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909476    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909478    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909481    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909484    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909487    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909489    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909492    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909494    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909497    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909499    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909502    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909505    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:13:24.912188 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909507    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909510    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909512    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909515    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909517    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909520    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909523    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909537    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909540    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909542    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909545    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909548    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909550    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909553    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909555    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909558    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909562    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909565    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909569    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909573    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:13:24.912712 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909576    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909578    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909581    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909584    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909586    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909589    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909591    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909594    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909596    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909599    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909602    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909605    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909607    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909610    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909613    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909615    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909618    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909621    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909623    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:13:24.913221 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909626    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909629    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909631    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909634    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909638    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909641    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909645    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909650    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909652    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909655    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909658    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909661    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909664    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909668    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:13:24.913697 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909671    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.909676    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909791    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909796    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909799    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909802    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909805    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909807    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909810    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909813    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909816    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909819    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909822    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909824    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909828    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:13:24.914060 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909830    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909833    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909836    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909839    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909842    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909845    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909847    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909850    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909852    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909855    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909858    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909862    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909865    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909867    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909870    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909873    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909875    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909878    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909880    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909883    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:13:24.914424 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909885    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909888    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909890    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909893    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909895    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909898    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909901    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909903    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909905    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909908    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909910    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909913    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909916    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909920    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909923    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909925    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909928    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909931    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909933    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:13:24.914951 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909936    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909938    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909941    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909944    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909947    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909949    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909952    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909954    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909957    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909960    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909962    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909965    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909968    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909970    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909973    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909975    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909978    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909980    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909983    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909986    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:13:24.915418 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909988    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909991    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909993    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909996    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.909998    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910001    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910004    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910006    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910009    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910011    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910015    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910019    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910021    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910024    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910028    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910030    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910033    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910036    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910038    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:13:24.915993 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910041    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910044    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910046    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910050    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910054    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910057    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910060    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:24.910063    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.910068    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.910176    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.912136    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.912967    2554 server.go:1019] "Starting client certificate rotation"
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.913061    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:13:24.916456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.913117    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:13:24.917700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.917677    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:13:24.922214 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.922190    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:13:24.933505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.933482    2554 log.go:25] "Validated CRI v1 runtime API"
Sep 10 17:13:24.937547 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.937406    2554 log.go:25] "Validated CRI v1 image API"
Sep 10 17:13:24.938435 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.938421    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 17:13:24.942126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.942108    2554 fs.go:135] Filesystem UUIDs: map[522f88a6-39fd-47e2-bffa-b19b64749039:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f901461d-f54f-4ba4-9e80-ee04b086e922:/dev/nvme0n1p4]
Sep 10 17:13:24.942225 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.942126    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 17:13:24.946947 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.946813    2554 manager.go:216] Machine: {Timestamp:2026-09-10 17:13:24.945812145 +0000 UTC m=+0.302302537 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3123044 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2614f9d5f5da5524c2d67e18b25b6d SystemUUID:ec2614f9-d5f5-da55-24c2-d67e18b25b6d BootID:5cae5ab9-a62e-46ee-946f-843a3fb6d427 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:65:56:e0:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:65:56:e0:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:76:2e:0c:95:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 17:13:24.946947 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.946935    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 17:13:24.947102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.947015    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 17:13:24.947168 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.947150    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:13:24.949696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.949665    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 17:13:24.949872 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.949701    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 17:13:24.950576 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.950565    2554 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 17:13:24.950642 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.950582    2554 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 17:13:24.950642 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.950592    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:13:24.951333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.951321    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:13:24.952988 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.952975    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 17:13:24.956134 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956120    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:13:24.956269 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956258    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 17:13:24.956325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956298    2554 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 17:13:24.956325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956313    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 17:13:24.956415 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956328    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 17:13:24.956415 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956342    2554 kubelet.go:398] "Adding apiserver pod source"
Sep 10 17:13:24.956415 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956355    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 17:13:24.956756 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956742    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:13:24.956831 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.956766    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:13:24.958909 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.958893    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 17:13:24.961450 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.961435    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 17:13:24.961512 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.961465    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 17:13:24.962029 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962017    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 17:13:24.962066 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962037    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 17:13:24.962066 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962043    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 17:13:24.962130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962072    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 17:13:24.962130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962081    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 17:13:24.962130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962087    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 17:13:24.962130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962127    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 17:13:24.962238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962132    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 17:13:24.962238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962139    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 17:13:24.962238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962145    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 17:13:24.962238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962153    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 17:13:24.962238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962160    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 17:13:24.962845 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962836    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 17:13:24.962845 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.962845    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 17:13:24.967139 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.967120    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 17:13:24.967221 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:24.967195    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 17:13:24.967343 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:24.967321    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 17:13:24.967375 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.967348    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2js58"
Sep 10 17:13:24.967518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.967507    2554 server.go:1268] "Started kubelet"
Sep 10 17:13:24.967659 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.967604    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 17:13:24.967786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.967623    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 17:13:24.967839 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.967812    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 17:13:24.968391 ip-10-0-142-52 systemd[1]: Started Kubernetes Kubelet.
Sep 10 17:13:24.969102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.969086    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 17:13:24.969667 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.969648    2554 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 17:13:24.973546 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:24.972576    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-52.ec2.internal.18d404f59fcd295c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-52.ec2.internal,UID:ip-10-0-142-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-52.ec2.internal,},FirstTimestamp:2026-09-10 17:13:24.967450972 +0000 UTC m=+0.323941365,LastTimestamp:2026-09-10 17:13:24.967450972 +0000 UTC m=+0.323941365,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-52.ec2.internal,}"
Sep 10 17:13:24.973658 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:24.973638    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 17:13:24.975550 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.975508    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 17:13:24.976040 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.976024    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 17:13:24.976126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.976060    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2js58"
Sep 10 17:13:24.976181 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:24.976123    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:24.976797 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.976763    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 17:13:24.976797 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.976766    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 17:13:24.976920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.976803    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 17:13:24.976920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.976888    2554 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 17:13:24.976920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.976903    2554 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 17:13:24.977190 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977176    2554 factory.go:55] Registering systemd factory
Sep 10 17:13:24.977190 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977191    2554 factory.go:223] Registration of the systemd container factory successfully
Sep 10 17:13:24.977409 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977394    2554 factory.go:153] Registering CRI-O factory
Sep 10 17:13:24.977459 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977412    2554 factory.go:223] Registration of the crio container factory successfully
Sep 10 17:13:24.977511 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977472    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 17:13:24.977511 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977489    2554 factory.go:103] Registering Raw factory
Sep 10 17:13:24.977511 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977504    2554 manager.go:1196] Started watching for new ooms in manager
Sep 10 17:13:24.977667 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:24.977619    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 17:13:24.978004 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.977989    2554 manager.go:349] Starting recovery of all containers
Sep 10 17:13:24.990851 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.990826    2554 manager.go:354] Recovery completed
Sep 10 17:13:24.991907 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:24.991886    2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-52.ec2.internal\" not found" node="ip-10-0-142-52.ec2.internal"
Sep 10 17:13:24.996411 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.996398    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:13:24.998238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.998219    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:13:24.998317 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.998257    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:13:24.998317 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.998273    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:13:24.998834 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.998821    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:13:24.998886 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.998824    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 17:13:24.998923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.998883    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 17:13:24.998923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.998905    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:13:24.999458 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.999443    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:13:24.999509 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.999467    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:13:24.999509 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:24.999478    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:13:25.000113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.000101    2554 policy_none.go:49] "None policy: Start"
Sep 10 17:13:25.000149 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.000118    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 17:13:25.000149 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.000129    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 17:13:25.001209 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.001196    2554 policy_none.go:47] "Start"
Sep 10 17:13:25.029094 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.029078    2554 manager.go:337] "Starting Device Plugin manager"
Sep 10 17:13:25.029185 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.029112    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 17:13:25.029185 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.029122    2554 server.go:87] "Starting device plugin registration server"
Sep 10 17:13:25.029450 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.029428    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 17:13:25.029661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.029449    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 17:13:25.029661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.029629    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 17:13:25.029848 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.029700    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 17:13:25.029848 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.029709    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 17:13:25.030618 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.030599    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 17:13:25.030743 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.030645    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.106685 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.106653    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 17:13:25.107826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.107802    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 17:13:25.107909 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.107853    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 17:13:25.107909 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.107875    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 17:13:25.108003 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.107916    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 17:13:25.110361 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.110340    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:13:25.130560 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.130542    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:13:25.131434 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.131416    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:13:25.131564 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.131449    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:13:25.131564 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.131462    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:13:25.131564 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.131483    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.139374 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.139359    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.139426 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.139379    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-52.ec2.internal\": node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.158193 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.158118    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.208514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.208493    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"]
Sep 10 17:13:25.208603 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.208576    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:13:25.209383 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.209362    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:13:25.209448 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.209395    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:13:25.209448 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.209408    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:13:25.210800 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.210787    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:13:25.210918 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.210904    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.210964 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.210934    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:13:25.211515 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.211502    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:13:25.211574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.211516    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:13:25.211574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.211524    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:13:25.211574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.211551    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:13:25.211574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.211556    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:13:25.211574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.211566    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:13:25.212620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.212604    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.212687 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.212633    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:13:25.213294 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.213279    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:13:25.213349 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.213304    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:13:25.213349 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.213316    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:13:25.241176 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.241153    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-52.ec2.internal\" not found" node="ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.245633 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.245618    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-52.ec2.internal\" not found" node="ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.258675 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.258648    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.278863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.278843    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f119684184aeb2e23bdcf787e218d99-config\") pod \"kube-apiserver-proxy-ip-10-0-142-52.ec2.internal\" (UID: \"8f119684184aeb2e23bdcf787e218d99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.278923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.278869    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.278923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.278890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.359407 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.359381    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.379785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.379750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f119684184aeb2e23bdcf787e218d99-config\") pod \"kube-apiserver-proxy-ip-10-0-142-52.ec2.internal\" (UID: \"8f119684184aeb2e23bdcf787e218d99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.379905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.379793    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.379905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.379818    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.379905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.379844    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f119684184aeb2e23bdcf787e218d99-config\") pod \"kube-apiserver-proxy-ip-10-0-142-52.ec2.internal\" (UID: \"8f119684184aeb2e23bdcf787e218d99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.379905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.379870    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.379905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.379873    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e4f9d586d66128144e4e5ad291a5b68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal\" (UID: \"7e4f9d586d66128144e4e5ad291a5b68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.460154 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.460102    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.544594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.544562    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.549518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.549503    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.560501 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.560484    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.662574 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.662523    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.765039 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.764954    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-52.ec2.internal\" not found"
Sep 10 17:13:25.832884 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.832855    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:13:25.877245 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.877222    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.887911 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.887888    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:13:25.889977 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.889963    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal"
Sep 10 17:13:25.901483 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.901464    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:13:25.912566 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.912550    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 17:13:25.912694 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.912675    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:13:25.912736 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.912692    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:13:25.957299 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.957270    2554 apiserver.go:52] "Watching apiserver"
Sep 10 17:13:25.975686 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.975637    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 17:13:25.976384 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.976360    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 17:13:25.976771 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.976748    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-n7w82","openshift-network-operator/iptables-alerter-rzk44","openshift-ovn-kubernetes/ovnkube-node-zw9fz","kube-system/global-pull-secret-syncer-499fh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm","openshift-cluster-node-tuning-operator/tuned-qlmk7","openshift-dns/node-resolver-ss6gj","openshift-image-registry/node-ca-zpxkk","openshift-multus/multus-additional-cni-plugins-dvwnv","openshift-network-diagnostics/network-check-target-q2mpq","kube-system/konnectivity-agent-qxtsv","kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal","openshift-multus/multus-xq984"]
Sep 10 17:13:25.978041 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.977990    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 17:08:25 +0000 UTC" deadline="2028-05-02 19:44:38.823740689 +0000 UTC"
Sep 10 17:13:25.978095 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.978041    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14402h31m12.84570275s"
Sep 10 17:13:25.978095 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.978040    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:25.979226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.979202    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xq984"
Sep 10 17:13:25.980177 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.980156    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:25.980836 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.980815    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 17:13:25.980932 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.980886    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 17:13:25.980932 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.980823    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bq56m\""
Sep 10 17:13:25.981054 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.980823    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 17:13:25.981255 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.981239    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:25.981339 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.981313    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:25.981478 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.981463    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 17:13:25.981730 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.981714    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e24a6e41-b03d-4e48-be14-b7487072c80b-host\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:25.981804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.981741    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e24a6e41-b03d-4e48-be14-b7487072c80b-serviceca\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:25.981804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.981768    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ktm\" (UniqueName: \"kubernetes.io/projected/e24a6e41-b03d-4e48-be14-b7487072c80b-kube-api-access-26ktm\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:25.982104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982087    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 17:13:25.982430 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982413    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 17:13:25.982491 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982421    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:25.982705 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982588    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-65ggg\""
Sep 10 17:13:25.982705 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982666    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 17:13:25.982866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982829    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8w2l\""
Sep 10 17:13:25.982957 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982873    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 17:13:25.982957 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.982873    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 17:13:25.983860 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.983835    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:25.985036 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.985015    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mkhpt\""
Sep 10 17:13:25.985163 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.985063    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 17:13:25.985163 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.985096    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:25.985284 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.985204    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:25.985765 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.985746    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 17:13:25.985859 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.985748    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 17:13:25.986270 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.986248    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:13:25.986377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.986308    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pqndg\""
Sep 10 17:13:25.986377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.986315    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 17:13:25.986651 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.986635    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:25.987301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.987282    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:13:25.988179 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.988163    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:25.989146 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989130    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 17:13:25.989315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989297    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 17:13:25.989360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989336    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 17:13:25.989401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989382    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2vj55\""
Sep 10 17:13:25.989449 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989399    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 17:13:25.989449 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989424    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 17:13:25.989519 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989491    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 17:13:25.989810 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.989794    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:25.989876 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:25.989857    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:25.990584 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.990556    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 17:13:25.990584 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.990577    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 17:13:25.990785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.990662    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-46qgl\""
Sep 10 17:13:25.991212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.991192    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:25.992620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.992588    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:25.993491 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.993472    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4cdcd\""
Sep 10 17:13:25.993673 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.993657    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 17:13:25.993820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.993805    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 17:13:25.995002 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.994985    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 17:13:25.995147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.995120    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6kn7s\""
Sep 10 17:13:25.995147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.995129    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 17:13:25.995857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:25.995370    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:13:26.015644 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.015583    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zk685"
Sep 10 17:13:26.019217 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.019199    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zk685"
Sep 10 17:13:26.078383 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.078172    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 17:13:26.082294 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082276    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-systemd-units\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.082353 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082303    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-slash\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.082353 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082322    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-cni-bin\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.082426 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082355    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:26.082426 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082395    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-dbus\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.082492 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082434    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e24a6e41-b03d-4e48-be14-b7487072c80b-host\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:26.082492 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082481    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-cni-multus\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.082555 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082513    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e24a6e41-b03d-4e48-be14-b7487072c80b-host\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:26.082590 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082574    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22d0f763-d51f-4444-89fd-0732b083afa3-hosts-file\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.082648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082597    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0f763-d51f-4444-89fd-0732b083afa3-tmp-dir\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.082648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082629    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-env-overrides\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.082756 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082655    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-cni-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.082756 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082711    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-cnibin\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.082836 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082754    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-systemd\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.082836 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082800    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-kubelet\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.082916 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-log-socket\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.082916 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082863    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75aa5079-794a-466b-ba77-8a88ce296e52-konnectivity-ca\") pod \"konnectivity-agent-qxtsv\" (UID: \"75aa5079-794a-466b-ba77-8a88ce296e52\") " pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:26.082916 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082896    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-kubelet-config\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.082916 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082915    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.083053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082931    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-system-cni-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.083053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082957    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-daemon-config\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.083053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.082973    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-multus-certs\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.083053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083005    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-cni-binary-copy\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.083053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.083053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083043    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-socket-dir-parent\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.083280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083063    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.083280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083080    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-run-netns\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.083280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-node-log\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.083280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083151    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-cni-netd\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.083280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083188    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ktm\" (UniqueName: \"kubernetes.io/projected/e24a6e41-b03d-4e48-be14-b7487072c80b-kube-api-access-26ktm\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:26.083280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083220    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.083490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-cni-bin\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.083490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083343    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-kubernetes\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.083490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083364    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-cnibin\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.083490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-registration-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.083490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083421    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-device-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.083490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083443    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.083490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083458    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ns6s\" (UniqueName: \"kubernetes.io/projected/f296dc1c-de2e-404f-b247-c939ec024330-kube-api-access-8ns6s\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-modprobe-d\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083639    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afe3cd64-a7b4-40fc-b93a-feac4151a367-tmp\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083674    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-var-lib-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083702    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysconfig\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083727    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5r9\" (UniqueName: \"kubernetes.io/projected/871f6bac-b621-4a96-8ccb-9156dd3f0ded-kube-api-access-kl5r9\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083752    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnj2z\" (UniqueName: \"kubernetes.io/projected/afe3cd64-a7b4-40fc-b93a-feac4151a367-kube-api-access-rnj2z\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083804    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e24a6e41-b03d-4e48-be14-b7487072c80b-serviceca\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083833    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f296dc1c-de2e-404f-b247-c939ec024330-iptables-alerter-script\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.083858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083858    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f296dc1c-de2e-404f-b247-c939ec024330-host-slash\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083884    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-etc-kubernetes\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083928    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-var-lib-kubelet\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.083966    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-host\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084005    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovnkube-script-lib\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084029    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-hostroot\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084050    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-sys\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084074    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-tuned\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-system-cni-dir\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084128    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcx4\" (UniqueName: \"kubernetes.io/projected/dd10cfa9-57c4-4113-9375-58f262337601-kube-api-access-zzcx4\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084166    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-systemd\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084188    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-netns\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084212    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-conf-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084249    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q622v\" (UniqueName: \"kubernetes.io/projected/22d0f763-d51f-4444-89fd-0732b083afa3-kube-api-access-q622v\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084270    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-socket-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.084303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084296    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovnkube-config\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084349    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4fx\" (UniqueName: \"kubernetes.io/projected/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-kube-api-access-ss4fx\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084391    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-ovn\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084414    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084458    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-k8s-cni-cncf-io\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084476    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysctl-d\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084500    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-run\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084555    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lfr\" (UniqueName: \"kubernetes.io/projected/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-kube-api-access-s2lfr\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084602    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-sys-fs\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75aa5079-794a-466b-ba77-8a88ce296e52-agent-certs\") pod \"konnectivity-agent-qxtsv\" (UID: \"75aa5079-794a-466b-ba77-8a88ce296e52\") " pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084639    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e24a6e41-b03d-4e48-be14-b7487072c80b-serviceca\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084643    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-etc-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084712    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysctl-conf\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084739    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-lib-modules\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.084816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084767    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovn-node-metrics-cert\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.085319 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084803    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:26.085319 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084827    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-cni-binary-copy\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.085319 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084842    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-kubelet\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.085319 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.084857    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8k4x\" (UniqueName: \"kubernetes.io/projected/873150c3-9650-4c7e-b4ba-5f44b32f2127-kube-api-access-d8k4x\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.090300 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.090273    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 17:13:26.094169 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.094148    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ktm\" (UniqueName: \"kubernetes.io/projected/e24a6e41-b03d-4e48-be14-b7487072c80b-kube-api-access-26ktm\") pod \"node-ca-zpxkk\" (UID: \"e24a6e41-b03d-4e48-be14-b7487072c80b\") " pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:26.185550 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185500    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.185550 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185532    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-run-netns\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-node-log\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-cni-netd\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185596    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-cni-bin\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-cni-bin\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185674    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-node-log\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185688    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-cni-netd\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185719    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-run-netns\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185722    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-kubernetes\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.185761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185754    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-cnibin\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185779    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-registration-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185801    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-device-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185824    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185831    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-cnibin\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185851    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ns6s\" (UniqueName: \"kubernetes.io/projected/f296dc1c-de2e-404f-b247-c939ec024330-kube-api-access-8ns6s\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-modprobe-d\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185879    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-registration-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185881    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-kubernetes\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185920    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-device-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185936    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-modprobe-d\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.185977    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afe3cd64-a7b4-40fc-b93a-feac4151a367-tmp\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-var-lib-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186056    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysconfig\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186081    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5r9\" (UniqueName: \"kubernetes.io/projected/871f6bac-b621-4a96-8ccb-9156dd3f0ded-kube-api-access-kl5r9\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186106    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnj2z\" (UniqueName: \"kubernetes.io/projected/afe3cd64-a7b4-40fc-b93a-feac4151a367-kube-api-access-rnj2z\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.186285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186134    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f296dc1c-de2e-404f-b247-c939ec024330-iptables-alerter-script\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186148    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysconfig\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186160    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f296dc1c-de2e-404f-b247-c939ec024330-host-slash\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186183    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-etc-kubernetes\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186199    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-var-lib-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186208    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-var-lib-kubelet\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186235    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-host\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186259    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovnkube-script-lib\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-hostroot\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-etc-kubernetes\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186308    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-sys\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186313    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-host\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186318    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-var-lib-kubelet\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186257    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186342    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f296dc1c-de2e-404f-b247-c939ec024330-host-slash\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-tuned\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186381    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-system-cni-dir\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186395    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-sys\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.187165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcx4\" (UniqueName: \"kubernetes.io/projected/dd10cfa9-57c4-4113-9375-58f262337601-kube-api-access-zzcx4\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186439    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-system-cni-dir\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186457    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-systemd\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186481    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-netns\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186492    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-conf-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186535    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-systemd\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186537    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q622v\" (UniqueName: \"kubernetes.io/projected/22d0f763-d51f-4444-89fd-0732b083afa3-kube-api-access-q622v\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186623    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-socket-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186654    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186684    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovnkube-config\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186726    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4fx\" (UniqueName: \"kubernetes.io/projected/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-kube-api-access-ss4fx\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-ovn\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186783    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f296dc1c-de2e-404f-b247-c939ec024330-iptables-alerter-script\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-netns\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.188100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186808    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-k8s-cni-cncf-io\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186816    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186869    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-conf-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186880    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186908    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysctl-d\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186919    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-k8s-cni-cncf-io\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186935    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-run\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.186935    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-run-ovn\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187054    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysctl-d\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187060    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187084    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-run\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187094    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-socket-dir\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187122    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lfr\" (UniqueName: \"kubernetes.io/projected/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-kube-api-access-s2lfr\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187149    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187170    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-sys-fs\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187195    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75aa5079-794a-466b-ba77-8a88ce296e52-agent-certs\") pod \"konnectivity-agent-qxtsv\" (UID: \"75aa5079-794a-466b-ba77-8a88ce296e52\") " pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187221    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-etc-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.189038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187245    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysctl-conf\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-lib-modules\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187295    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovn-node-metrics-cert\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187301    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-etc-openvswitch\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187316    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/873150c3-9650-4c7e-b4ba-5f44b32f2127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187322    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187356    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-cni-binary-copy\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-kubelet\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8k4x\" (UniqueName: \"kubernetes.io/projected/873150c3-9650-4c7e-b4ba-5f44b32f2127-kube-api-access-d8k4x\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187433    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-lib-modules\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187483    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-systemd-units\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187270    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-sys-fs\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187592    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovnkube-script-lib\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187602    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-sysctl-conf\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187435    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-systemd-units\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187686    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-kubelet\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187731    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-hostroot\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.190544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187793    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-slash\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187822    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-cni-bin\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187849    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-dbus\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-cni-multus\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187939    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-cni-bin\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187958    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22d0f763-d51f-4444-89fd-0732b083afa3-hosts-file\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187982    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0f763-d51f-4444-89fd-0732b083afa3-tmp-dir\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.187986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-slash\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188005    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-env-overrides\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188030    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-cni-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188048    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-cni-binary-copy\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.188074    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188083    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-dbus\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188111    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-cnibin\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188110    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22d0f763-d51f-4444-89fd-0732b083afa3-hosts-file\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188054    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-cnibin\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.188149    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:26.688127896 +0000 UTC m=+2.044618301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:26.191453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188182    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-cni-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188234    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-systemd\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188270    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-kubelet\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188296    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-log-socket\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188322    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75aa5079-794a-466b-ba77-8a88ce296e52-konnectivity-ca\") pod \"konnectivity-agent-qxtsv\" (UID: \"75aa5079-794a-466b-ba77-8a88ce296e52\") " pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188349    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-kubelet-config\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188366    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-systemd\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188376    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188145    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-var-lib-cni-multus\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188415    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-system-cni-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188426    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-host-kubelet\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-daemon-config\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188473    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/871f6bac-b621-4a96-8ccb-9156dd3f0ded-log-socket\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188482    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-env-overrides\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188502    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d0f763-d51f-4444-89fd-0732b083afa3-tmp-dir\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.188548    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188588    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-system-cni-dir\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.192445 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.188597    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret podName:4ff0d9fd-0612-4d96-b385-235ce5bcbed0 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:26.68858056 +0000 UTC m=+2.045070959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret") pod "global-pull-secret-syncer-499fh" (UID: "4ff0d9fd-0612-4d96-b385-235ce5bcbed0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-multus-certs\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188545    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-kubelet-config\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188654    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-host-run-multus-certs\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188673    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-cni-binary-copy\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188703    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-socket-dir-parent\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188807    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-socket-dir-parent\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.188824    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd10cfa9-57c4-4113-9375-58f262337601-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.189360    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/873150c3-9650-4c7e-b4ba-5f44b32f2127-cni-binary-copy\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.189372    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovnkube-config\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.189429    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-multus-daemon-config\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.189643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/75aa5079-794a-466b-ba77-8a88ce296e52-konnectivity-ca\") pod \"konnectivity-agent-qxtsv\" (UID: \"75aa5079-794a-466b-ba77-8a88ce296e52\") " pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.189867    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afe3cd64-a7b4-40fc-b93a-feac4151a367-tmp\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.190459    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afe3cd64-a7b4-40fc-b93a-feac4151a367-etc-tuned\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.190667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/75aa5079-794a-466b-ba77-8a88ce296e52-agent-certs\") pod \"konnectivity-agent-qxtsv\" (UID: \"75aa5079-794a-466b-ba77-8a88ce296e52\") " pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:26.193404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.190875    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/871f6bac-b621-4a96-8ccb-9156dd3f0ded-ovn-node-metrics-cert\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.199964 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.199939    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:13:26.200055 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.199973    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:13:26.200055 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.199989    2554 projected.go:196] Error preparing data for projected volume kube-api-access-97xrt for pod openshift-network-diagnostics/network-check-target-q2mpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:26.200122 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.200069    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt podName:00ad883b-3f74-42a9-957a-c6369c41581b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:26.70004971 +0000 UTC m=+2.056540131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-97xrt" (UniqueName: "kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt") pod "network-check-target-q2mpq" (UID: "00ad883b-3f74-42a9-957a-c6369c41581b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:26.200831 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.200738    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ns6s\" (UniqueName: \"kubernetes.io/projected/f296dc1c-de2e-404f-b247-c939ec024330-kube-api-access-8ns6s\") pod \"iptables-alerter-rzk44\" (UID: \"f296dc1c-de2e-404f-b247-c939ec024330\") " pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.204410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.201848    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnj2z\" (UniqueName: \"kubernetes.io/projected/afe3cd64-a7b4-40fc-b93a-feac4151a367-kube-api-access-rnj2z\") pod \"tuned-qlmk7\" (UID: \"afe3cd64-a7b4-40fc-b93a-feac4151a367\") " pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.204410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.201943    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5r9\" (UniqueName: \"kubernetes.io/projected/871f6bac-b621-4a96-8ccb-9156dd3f0ded-kube-api-access-kl5r9\") pod \"ovnkube-node-zw9fz\" (UID: \"871f6bac-b621-4a96-8ccb-9156dd3f0ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.204410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.202911    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8k4x\" (UniqueName: \"kubernetes.io/projected/873150c3-9650-4c7e-b4ba-5f44b32f2127-kube-api-access-d8k4x\") pod \"multus-additional-cni-plugins-dvwnv\" (UID: \"873150c3-9650-4c7e-b4ba-5f44b32f2127\") " pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.204410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.203710    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4fx\" (UniqueName: \"kubernetes.io/projected/1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad-kube-api-access-ss4fx\") pod \"multus-xq984\" (UID: \"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad\") " pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.204410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.203957    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcx4\" (UniqueName: \"kubernetes.io/projected/dd10cfa9-57c4-4113-9375-58f262337601-kube-api-access-zzcx4\") pod \"aws-ebs-csi-driver-node-gsjvm\" (UID: \"dd10cfa9-57c4-4113-9375-58f262337601\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.204410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.204174    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lfr\" (UniqueName: \"kubernetes.io/projected/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-kube-api-access-s2lfr\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:26.204410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.204292    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q622v\" (UniqueName: \"kubernetes.io/projected/22d0f763-d51f-4444-89fd-0732b083afa3-kube-api-access-q622v\") pod \"node-resolver-ss6gj\" (UID: \"22d0f763-d51f-4444-89fd-0732b083afa3\") " pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.212338 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.212308    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4f9d586d66128144e4e5ad291a5b68.slice/crio-aa136a10d44e9913184b1c8b6a4e4713231624189b1637456d18af08bc851b17 WatchSource:0}: Error finding container aa136a10d44e9913184b1c8b6a4e4713231624189b1637456d18af08bc851b17: Status 404 returned error can't find the container with id aa136a10d44e9913184b1c8b6a4e4713231624189b1637456d18af08bc851b17
Sep 10 17:13:26.212591 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.212568    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f119684184aeb2e23bdcf787e218d99.slice/crio-249f3b93b5422c0e36c6c11967e87935ff1113438420ff2a16445e7a2bf35b82 WatchSource:0}: Error finding container 249f3b93b5422c0e36c6c11967e87935ff1113438420ff2a16445e7a2bf35b82: Status 404 returned error can't find the container with id 249f3b93b5422c0e36c6c11967e87935ff1113438420ff2a16445e7a2bf35b82
Sep 10 17:13:26.218440 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.218414    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:13:26.328555 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.328439    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zpxkk"
Sep 10 17:13:26.338631 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.338593    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xq984"
Sep 10 17:13:26.338788 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.338767    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24a6e41_b03d_4e48_be14_b7487072c80b.slice/crio-cc86228de753cdafa3e9900064acf2beb329c904f7e43a64f03629b5bb836d0c WatchSource:0}: Error finding container cc86228de753cdafa3e9900064acf2beb329c904f7e43a64f03629b5bb836d0c: Status 404 returned error can't find the container with id cc86228de753cdafa3e9900064acf2beb329c904f7e43a64f03629b5bb836d0c
Sep 10 17:13:26.347276 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.347255    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec44f3f_d2d1_4e65_bd8b_8af8a81951ad.slice/crio-2e3376b76fbfb2037a79d582911dbd26ef9328b0cc8ba3bf3fe4280edd8a3d21 WatchSource:0}: Error finding container 2e3376b76fbfb2037a79d582911dbd26ef9328b0cc8ba3bf3fe4280edd8a3d21: Status 404 returned error can't find the container with id 2e3376b76fbfb2037a79d582911dbd26ef9328b0cc8ba3bf3fe4280edd8a3d21
Sep 10 17:13:26.351513 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.351496    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ss6gj"
Sep 10 17:13:26.355500 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.355478    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm"
Sep 10 17:13:26.360004 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.359981    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d0f763_d51f_4444_89fd_0732b083afa3.slice/crio-54451a3132ce7c6c1d967a3655f36d40078333ee879dfad39c253a2e6ee4dfca WatchSource:0}: Error finding container 54451a3132ce7c6c1d967a3655f36d40078333ee879dfad39c253a2e6ee4dfca: Status 404 returned error can't find the container with id 54451a3132ce7c6c1d967a3655f36d40078333ee879dfad39c253a2e6ee4dfca
Sep 10 17:13:26.365360 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.365341    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd10cfa9_57c4_4113_9375_58f262337601.slice/crio-6fcdfe005cb74ed5c5725e44a07c75f616f8d311a938a45198b7fef8db5c6fff WatchSource:0}: Error finding container 6fcdfe005cb74ed5c5725e44a07c75f616f8d311a938a45198b7fef8db5c6fff: Status 404 returned error can't find the container with id 6fcdfe005cb74ed5c5725e44a07c75f616f8d311a938a45198b7fef8db5c6fff
Sep 10 17:13:26.368705 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.368681    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlmk7"
Sep 10 17:13:26.377674 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.377651    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe3cd64_a7b4_40fc_b93a_feac4151a367.slice/crio-320862de7b05c814344f653d05f5770d8ba3d89d468a0378f5b00f0b395e29f3 WatchSource:0}: Error finding container 320862de7b05c814344f653d05f5770d8ba3d89d468a0378f5b00f0b395e29f3: Status 404 returned error can't find the container with id 320862de7b05c814344f653d05f5770d8ba3d89d468a0378f5b00f0b395e29f3
Sep 10 17:13:26.386500 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.386483    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:26.393262 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.393226    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dvwnv"
Sep 10 17:13:26.394632 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.394600    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871f6bac_b621_4a96_8ccb_9156dd3f0ded.slice/crio-5c94c89689070c9543a0e0c59057d6a225addc5d6023f32fb3ffdd86140c9c98 WatchSource:0}: Error finding container 5c94c89689070c9543a0e0c59057d6a225addc5d6023f32fb3ffdd86140c9c98: Status 404 returned error can't find the container with id 5c94c89689070c9543a0e0c59057d6a225addc5d6023f32fb3ffdd86140c9c98
Sep 10 17:13:26.402298 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.402277    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873150c3_9650_4c7e_b4ba_5f44b32f2127.slice/crio-9c712fa9a62779e0b14523edcb674169bd01f998007af43883060311c9789858 WatchSource:0}: Error finding container 9c712fa9a62779e0b14523edcb674169bd01f998007af43883060311c9789858: Status 404 returned error can't find the container with id 9c712fa9a62779e0b14523edcb674169bd01f998007af43883060311c9789858
Sep 10 17:13:26.411416 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.411396    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:26.416316 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.416294    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rzk44"
Sep 10 17:13:26.420559 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.420536    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75aa5079_794a_466b_ba77_8a88ce296e52.slice/crio-1c191f85c4fb4f021adb89746118b6817c88dc3b19ec94a0fe8700d3e5e11f5d WatchSource:0}: Error finding container 1c191f85c4fb4f021adb89746118b6817c88dc3b19ec94a0fe8700d3e5e11f5d: Status 404 returned error can't find the container with id 1c191f85c4fb4f021adb89746118b6817c88dc3b19ec94a0fe8700d3e5e11f5d
Sep 10 17:13:26.424906 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:26.424886    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf296dc1c_de2e_404f_b247_c939ec024330.slice/crio-02915bd6abe9af045250b3b78527c6d97a3024f30c9473f1b875c4a87362920b WatchSource:0}: Error finding container 02915bd6abe9af045250b3b78527c6d97a3024f30c9473f1b875c4a87362920b: Status 404 returned error can't find the container with id 02915bd6abe9af045250b3b78527c6d97a3024f30c9473f1b875c4a87362920b
Sep 10 17:13:26.473322 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.473296    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:13:26.692661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.692543    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:26.692661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.692622    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:26.692904 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.692669    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:26.692904 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.692719    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:26.692904 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.692762    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:27.692738611 +0000 UTC m=+3.049229006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:26.692904 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.692785    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret podName:4ff0d9fd-0612-4d96-b385-235ce5bcbed0 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:27.692772302 +0000 UTC m=+3.049262687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret") pod "global-pull-secret-syncer-499fh" (UID: "4ff0d9fd-0612-4d96-b385-235ce5bcbed0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:26.797568 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.797508    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:26.797781 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.797753    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:13:26.797781 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.797779    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:13:26.797915 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.797792    2554 projected.go:196] Error preparing data for projected volume kube-api-access-97xrt for pod openshift-network-diagnostics/network-check-target-q2mpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:26.797915 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:26.797856    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt podName:00ad883b-3f74-42a9-957a-c6369c41581b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:27.797837909 +0000 UTC m=+3.154328314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xrt" (UniqueName: "kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt") pod "network-check-target-q2mpq" (UID: "00ad883b-3f74-42a9-957a-c6369c41581b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:26.922353 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:26.920846    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:13:27.023139 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.022993    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:08:26 +0000 UTC" deadline="2028-02-07 23:51:03.64084411 +0000 UTC"
Sep 10 17:13:27.023139 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.023067    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12366h37m36.617781132s"
Sep 10 17:13:27.103342 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.103134    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:13:27.143162 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.143053    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal" event={"ID":"8f119684184aeb2e23bdcf787e218d99","Type":"ContainerStarted","Data":"249f3b93b5422c0e36c6c11967e87935ff1113438420ff2a16445e7a2bf35b82"}
Sep 10 17:13:27.165714 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.165672    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rzk44" event={"ID":"f296dc1c-de2e-404f-b247-c939ec024330","Type":"ContainerStarted","Data":"02915bd6abe9af045250b3b78527c6d97a3024f30c9473f1b875c4a87362920b"}
Sep 10 17:13:27.188180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.188140    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxtsv" event={"ID":"75aa5079-794a-466b-ba77-8a88ce296e52","Type":"ContainerStarted","Data":"1c191f85c4fb4f021adb89746118b6817c88dc3b19ec94a0fe8700d3e5e11f5d"}
Sep 10 17:13:27.212758 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.212525    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlmk7" event={"ID":"afe3cd64-a7b4-40fc-b93a-feac4151a367","Type":"ContainerStarted","Data":"320862de7b05c814344f653d05f5770d8ba3d89d468a0378f5b00f0b395e29f3"}
Sep 10 17:13:27.230759 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.230723    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zpxkk" event={"ID":"e24a6e41-b03d-4e48-be14-b7487072c80b","Type":"ContainerStarted","Data":"cc86228de753cdafa3e9900064acf2beb329c904f7e43a64f03629b5bb836d0c"}
Sep 10 17:13:27.245400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.245308    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" event={"ID":"7e4f9d586d66128144e4e5ad291a5b68","Type":"ContainerStarted","Data":"aa136a10d44e9913184b1c8b6a4e4713231624189b1637456d18af08bc851b17"}
Sep 10 17:13:27.269363 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.269257    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerStarted","Data":"9c712fa9a62779e0b14523edcb674169bd01f998007af43883060311c9789858"}
Sep 10 17:13:27.283470 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.283391    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"5c94c89689070c9543a0e0c59057d6a225addc5d6023f32fb3ffdd86140c9c98"}
Sep 10 17:13:27.292211 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.292110    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm" event={"ID":"dd10cfa9-57c4-4113-9375-58f262337601","Type":"ContainerStarted","Data":"6fcdfe005cb74ed5c5725e44a07c75f616f8d311a938a45198b7fef8db5c6fff"}
Sep 10 17:13:27.308457 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.308423    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ss6gj" event={"ID":"22d0f763-d51f-4444-89fd-0732b083afa3","Type":"ContainerStarted","Data":"54451a3132ce7c6c1d967a3655f36d40078333ee879dfad39c253a2e6ee4dfca"}
Sep 10 17:13:27.310748 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.310702    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xq984" event={"ID":"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad","Type":"ContainerStarted","Data":"2e3376b76fbfb2037a79d582911dbd26ef9328b0cc8ba3bf3fe4280edd8a3d21"}
Sep 10 17:13:27.705104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.705020    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:27.705104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.705063    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:27.705347 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.705205    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:27.705347 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.705266    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret podName:4ff0d9fd-0612-4d96-b385-235ce5bcbed0 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:29.705246934 +0000 UTC m=+5.061737339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret") pod "global-pull-secret-syncer-499fh" (UID: "4ff0d9fd-0612-4d96-b385-235ce5bcbed0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:27.705698 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.705678    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:27.705777 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.705746    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:29.705727937 +0000 UTC m=+5.062218336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:27.805779 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:27.805740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:27.806014 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.805980    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:13:27.806014 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.806007    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:13:27.806151 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.806020    2554 projected.go:196] Error preparing data for projected volume kube-api-access-97xrt for pod openshift-network-diagnostics/network-check-target-q2mpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:27.806151 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:27.806080    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt podName:00ad883b-3f74-42a9-957a-c6369c41581b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:29.806061157 +0000 UTC m=+5.162551543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xrt" (UniqueName: "kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt") pod "network-check-target-q2mpq" (UID: "00ad883b-3f74-42a9-957a-c6369c41581b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:28.023790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:28.023696    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:08:26 +0000 UTC" deadline="2028-06-24 12:50:20.838986314 +0000 UTC"
Sep 10 17:13:28.023790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:28.023740    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15667h36m52.815251527s"
Sep 10 17:13:28.095912 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:28.095569    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:13:28.109216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:28.108316    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:28.109216 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:28.108451    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:28.109216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:28.108923    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:28.109216 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:28.109034    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:28.109216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:28.109097    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:28.109216 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:28.109176    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:29.721497 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:29.721456    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:29.722016 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:29.721579    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:29.722016 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.721714    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:29.722016 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.721781    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:33.721761193 +0000 UTC m=+9.078251581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:29.722239 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.722216    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:29.722310 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.722276    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret podName:4ff0d9fd-0612-4d96-b385-235ce5bcbed0 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:33.722260791 +0000 UTC m=+9.078751176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret") pod "global-pull-secret-syncer-499fh" (UID: "4ff0d9fd-0612-4d96-b385-235ce5bcbed0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:29.822882 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:29.822841    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:29.823060 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.823029    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:13:29.823060 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.823050    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:13:29.823180 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.823063    2554 projected.go:196] Error preparing data for projected volume kube-api-access-97xrt for pod openshift-network-diagnostics/network-check-target-q2mpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:29.823180 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:29.823126    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt podName:00ad883b-3f74-42a9-957a-c6369c41581b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:33.823107405 +0000 UTC m=+9.179597792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xrt" (UniqueName: "kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt") pod "network-check-target-q2mpq" (UID: "00ad883b-3f74-42a9-957a-c6369c41581b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:30.108423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:30.108099    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:30.108423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:30.108151    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:30.108423 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:30.108243    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:30.108423 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:30.108320    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:30.108423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:30.108338    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:30.108423 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:30.108422    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:32.108725 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:32.108686    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:32.109375 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:32.108828    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:32.109375 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:32.109249    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:32.109375 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:32.109357    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:32.109652 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:32.109422    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:32.109652 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:32.109499    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:33.757481 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:33.757447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:33.758108 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:33.757500    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:33.758108 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.757704    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:33.758108 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.757806    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret podName:4ff0d9fd-0612-4d96-b385-235ce5bcbed0 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:41.75778547 +0000 UTC m=+17.114275865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret") pod "global-pull-secret-syncer-499fh" (UID: "4ff0d9fd-0612-4d96-b385-235ce5bcbed0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:33.758108 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.757994    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:33.758108 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.758052    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:41.758035499 +0000 UTC m=+17.114525896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:33.858043 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:33.858005    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:33.858525 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.858249    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:13:33.858525 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.858274    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:13:33.858525 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.858287    2554 projected.go:196] Error preparing data for projected volume kube-api-access-97xrt for pod openshift-network-diagnostics/network-check-target-q2mpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:33.858525 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:33.858345    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt podName:00ad883b-3f74-42a9-957a-c6369c41581b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:41.858324952 +0000 UTC m=+17.214815333 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xrt" (UniqueName: "kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt") pod "network-check-target-q2mpq" (UID: "00ad883b-3f74-42a9-957a-c6369c41581b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:34.108660 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:34.108478    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:34.108660 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:34.108518    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:34.108660 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:34.108625    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:34.108660 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:34.108478    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:34.109036 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:34.108762    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:34.109036 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:34.108858    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:36.108998 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:36.108964    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:36.108998 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:36.109006    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:36.109625 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:36.108964    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:36.109625 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:36.109119    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:36.109625 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:36.109224    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:36.109625 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:36.109313    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:38.108122 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:38.108094    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:38.108580 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:38.108099    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:38.108580 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:38.108203    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:38.108580 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:38.108286    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:38.108580 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:38.108101    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:38.108580 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:38.108376    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:40.109117 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:40.109078    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:40.109671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:40.109133    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:40.109671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:40.109099    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:40.109671 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:40.109222    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:40.109671 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:40.109313    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:40.109671 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:40.109410    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:41.811586 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:41.811542    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:41.812013 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:41.811604    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:41.812013 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.811633    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:41.812013 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.811704    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:57.81168878 +0000 UTC m=+33.168179174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:41.812013 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.811734    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:41.812013 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.811788    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret podName:4ff0d9fd-0612-4d96-b385-235ce5bcbed0 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:57.811772789 +0000 UTC m=+33.168263184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret") pod "global-pull-secret-syncer-499fh" (UID: "4ff0d9fd-0612-4d96-b385-235ce5bcbed0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:41.912792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:41.912771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:41.912949 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.912930    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:13:41.913017 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.912955    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:13:41.913017 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.912969    2554 projected.go:196] Error preparing data for projected volume kube-api-access-97xrt for pod openshift-network-diagnostics/network-check-target-q2mpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:41.913158 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:41.913028    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt podName:00ad883b-3f74-42a9-957a-c6369c41581b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:57.913008892 +0000 UTC m=+33.269499288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xrt" (UniqueName: "kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt") pod "network-check-target-q2mpq" (UID: "00ad883b-3f74-42a9-957a-c6369c41581b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:42.108901 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:42.108820    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:42.109057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:42.108939    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:42.109057 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:42.108947    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:42.109057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:42.108967    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:42.109057 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:42.109021    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:42.109255 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:42.109103    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:44.109007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:44.108971    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:44.109542 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:44.108971    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:44.109542 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:44.109103    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:44.109542 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:44.109214    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:44.109542 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:44.108971    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:44.109542 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:44.109330    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:46.108419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.108254    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:46.108891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.108285    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:46.108891 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:46.108487    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:46.108891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.108341    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:46.108891 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:46.108583    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:46.108891 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:46.108674    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:46.376895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.376629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"df584c5908551563628de42a209eb44bff4b933a129385669edd145442cae1cc"}
Sep 10 17:13:46.376895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.376830    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"d6b89959c5eb19732e9af03b3cf9bd4765fbd1b74d49acaf4996d8b1ed4d84d0"}
Sep 10 17:13:46.376895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.376860    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"e46ca609722f27fd12fc1512734ac8cd60c255b52cac384ec2b0f2aaf9b98d8d"}
Sep 10 17:13:46.379322 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.378892    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xq984" event={"ID":"1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad","Type":"ContainerStarted","Data":"9afe5200cad42ca097c848327e029ad3d19ac8381c1cdc45efd0a3585ec6be4b"}
Sep 10 17:13:46.380691 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.380522    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal" event={"ID":"8f119684184aeb2e23bdcf787e218d99","Type":"ContainerStarted","Data":"05fab82db4c3407f70ab9ed65ef185b390d268feadc3e0c7937ae2b6b7d50616"}
Sep 10 17:13:46.382100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.382071    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlmk7" event={"ID":"afe3cd64-a7b4-40fc-b93a-feac4151a367","Type":"ContainerStarted","Data":"bae3e1ce1f16eaf0afa8b12d8a279cc763c18e67b9a674826f140d48877de139"}
Sep 10 17:13:46.395828 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.395626    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xq984" podStartSLOduration=1.786967685 podStartE2EDuration="21.395606624s" totalImagesPullingTime="19.608638939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:46.395052787 +0000 UTC m=+21.751543188" watchObservedRunningTime="2026-09-10 17:13:46.395606624 +0000 UTC m=+21.752097025"
Sep 10 17:13:46.408375 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.408333    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-52.ec2.internal" podStartSLOduration=20.408319098 podStartE2EDuration="20.408319098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:46.407983497 +0000 UTC m=+21.764473896" watchObservedRunningTime="2026-09-10 17:13:46.408319098 +0000 UTC m=+21.764809499"
Sep 10 17:13:46.429290 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:46.427952    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qlmk7" podStartSLOduration=1.138145838 podStartE2EDuration="20.427933609s" totalImagesPullingTime="19.289787771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:46.427165036 +0000 UTC m=+21.783655436" watchObservedRunningTime="2026-09-10 17:13:46.427933609 +0000 UTC m=+21.784424022"
Sep 10 17:13:47.293639 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.293618    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 17:13:47.385460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.385427    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxtsv" event={"ID":"75aa5079-794a-466b-ba77-8a88ce296e52","Type":"ContainerStarted","Data":"42cbce88c2481a4585b67fc8cebd6ff14931b6074f910b0b9e8212f0c45cd91a"}
Sep 10 17:13:47.386976 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.386951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zpxkk" event={"ID":"e24a6e41-b03d-4e48-be14-b7487072c80b","Type":"ContainerStarted","Data":"5224c5073e7314cc868934743f2dbf3944a9f8686832d4c5487f0114bdbd251e"}
Sep 10 17:13:47.388497 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.388469    2554 generic.go:356] "Generic (PLEG): container finished" podID="7e4f9d586d66128144e4e5ad291a5b68" containerID="1d87929ff8974991b37d8fbfa45b0791afac28faa22683eb95b89e2a096b1b7e" exitCode=0
Sep 10 17:13:47.388588 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.388560    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" event={"ID":"7e4f9d586d66128144e4e5ad291a5b68","Type":"ContainerDied","Data":"1d87929ff8974991b37d8fbfa45b0791afac28faa22683eb95b89e2a096b1b7e"}
Sep 10 17:13:47.390070 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.390050    2554 generic.go:356] "Generic (PLEG): container finished" podID="873150c3-9650-4c7e-b4ba-5f44b32f2127" containerID="046e89795ee3d2ee19f8dab663578771bee121868754cfd59d4dff3efca8ac3c" exitCode=0
Sep 10 17:13:47.390147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.390117    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerDied","Data":"046e89795ee3d2ee19f8dab663578771bee121868754cfd59d4dff3efca8ac3c"}
Sep 10 17:13:47.393492 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.393475    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"3ffa1378b4997a76641f2ad4291102bab2744a50ed990f4c04c1b697bbae1a28"}
Sep 10 17:13:47.393601 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.393551    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"705703d064ca6ffb2ed4b318216d472671d10f50ea2c190c796b9653ca3643a6"}
Sep 10 17:13:47.393663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.393600    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"5e3c000d80a6d2f9efa0ff31eb924450874fc4d9e39fca82d656b4e2555e0863"}
Sep 10 17:13:47.395352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.395328    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm" event={"ID":"dd10cfa9-57c4-4113-9375-58f262337601","Type":"ContainerStarted","Data":"bc88aa2f263ef1edda0d31a4969972b1ddced72c45e472c34e4de0b048e16eac"}
Sep 10 17:13:47.395422 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.395372    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm" event={"ID":"dd10cfa9-57c4-4113-9375-58f262337601","Type":"ContainerStarted","Data":"57606590ef6d8d0133b08212700f65ba4f64cb8f94dfda261902c2b1d888d263"}
Sep 10 17:13:47.397029 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.397004    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ss6gj" event={"ID":"22d0f763-d51f-4444-89fd-0732b083afa3","Type":"ContainerStarted","Data":"361178d252e61967c88843710a6f6a3679a6139b41e0b9882c3dfcd6bc71defb"}
Sep 10 17:13:47.398692 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.398674    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rzk44" event={"ID":"f296dc1c-de2e-404f-b247-c939ec024330","Type":"ContainerStarted","Data":"ebd6758eececd67bac735f8e9a4249cc3fc93020cd233c3b5f04bf9ef37d28d0"}
Sep 10 17:13:47.414715 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.414671    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qxtsv" podStartSLOduration=2.1844541299999998 podStartE2EDuration="21.414655497s" totalImagesPullingTime="19.230201367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:47.400727097 +0000 UTC m=+22.757217496" watchObservedRunningTime="2026-09-10 17:13:47.414655497 +0000 UTC m=+22.771145899"
Sep 10 17:13:47.415237 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.415211    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zpxkk" podStartSLOduration=3.089345535 podStartE2EDuration="22.415201765s" totalImagesPullingTime="19.32585623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:47.414475524 +0000 UTC m=+22.770965924" watchObservedRunningTime="2026-09-10 17:13:47.415201765 +0000 UTC m=+22.771692167"
Sep 10 17:13:47.451211 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.451160    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rzk44" podStartSLOduration=2.211088852 podStartE2EDuration="21.451141819s" totalImagesPullingTime="19.240052967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:47.450928086 +0000 UTC m=+22.807418487" watchObservedRunningTime="2026-09-10 17:13:47.451141819 +0000 UTC m=+22.807632220"
Sep 10 17:13:47.483013 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:47.482783    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ss6gj" podStartSLOduration=2.192759724 podStartE2EDuration="21.482767835s" totalImagesPullingTime="19.290008111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:47.482168407 +0000 UTC m=+22.838658808" watchObservedRunningTime="2026-09-10 17:13:47.482767835 +0000 UTC m=+22.839258235"
Sep 10 17:13:48.044716 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.044600    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T17:13:47.293635131Z","UUID":"6f6c99db-c4a1-4da7-b3e2-6eb1e7b32495","Handler":null,"Name":"","Endpoint":""}
Sep 10 17:13:48.046401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.046377    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 17:13:48.046525 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.046412    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 17:13:48.108349 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.108289    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:48.108519 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.108289    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:48.108519 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:48.108467    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:48.108655 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:48.108542    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:48.108655 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.108288    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:48.108655 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:48.108629    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:48.310401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.310317    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:48.310993 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.310971    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:48.402831 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.402785    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" event={"ID":"7e4f9d586d66128144e4e5ad291a5b68","Type":"ContainerStarted","Data":"ea1b0d99decbcfa673085f928a2bb9429d72de529d848e32763d99e6ff254f52"}
Sep 10 17:13:48.405352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.404986    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm" event={"ID":"dd10cfa9-57c4-4113-9375-58f262337601","Type":"ContainerStarted","Data":"d6fee2942ed6f445bbf0d5244ea32d20d21f27450b249ea8a9c73898014fa1ae"}
Sep 10 17:13:48.405352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.405264    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:48.405783 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.405758    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qxtsv"
Sep 10 17:13:48.420038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.419989    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-52.ec2.internal" podStartSLOduration=22.419975435 podStartE2EDuration="22.419975435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:48.419695836 +0000 UTC m=+23.776186237" watchObservedRunningTime="2026-09-10 17:13:48.419975435 +0000 UTC m=+23.776465835"
Sep 10 17:13:48.454904 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:48.454851    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsjvm" podStartSLOduration=1.5319515080000001 podStartE2EDuration="22.454832169s" totalImagesPullingTime="20.922880661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:48.438581803 +0000 UTC m=+23.795072206" watchObservedRunningTime="2026-09-10 17:13:48.454832169 +0000 UTC m=+23.811322571"
Sep 10 17:13:49.410829 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:49.410783    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"35a501c149a9d865e4c56719ebfd617a505996a8ac31e2cb4d5b8dc809698eed"}
Sep 10 17:13:50.109023 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:50.108995    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:50.109191 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:50.109109    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:50.109191 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:50.109148    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:50.109191 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:50.109108    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:50.109351 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:50.109220    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:50.109351 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:50.109283    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:52.108906 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.108662    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:52.109292 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.108709    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:52.109292 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:52.108933    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:52.109292 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:52.109028    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:52.109292 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.108710    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:52.109292 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:52.109137    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:52.418638 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.418555    2554 generic.go:356] "Generic (PLEG): container finished" podID="873150c3-9650-4c7e-b4ba-5f44b32f2127" containerID="5c5a37ed450827368bdd7b8ad5801f5aa733398e6ba648617dde5ea063af8c1f" exitCode=0
Sep 10 17:13:52.418775 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.418639    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerDied","Data":"5c5a37ed450827368bdd7b8ad5801f5aa733398e6ba648617dde5ea063af8c1f"}
Sep 10 17:13:52.422446 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.422422    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" event={"ID":"871f6bac-b621-4a96-8ccb-9156dd3f0ded","Type":"ContainerStarted","Data":"bf8ee62d166452376271d86e1d38569ee157356bfb3a0517613895b6ad8c3ce8"}
Sep 10 17:13:52.422752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.422731    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:52.422841 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.422798    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:52.422900 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.422843    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:52.437690 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.437669    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:52.437779 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.437769    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:13:52.462579 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:52.462518    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz" podStartSLOduration=6.745360612 podStartE2EDuration="26.462504934s" totalImagesPullingTime="19.717144322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:52.462456955 +0000 UTC m=+27.818947355" watchObservedRunningTime="2026-09-10 17:13:52.462504934 +0000 UTC m=+27.818995330"
Sep 10 17:13:53.302834 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.302627    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-499fh"]
Sep 10 17:13:53.303511 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.302945    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:53.303511 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:53.303044    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:53.305726 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.305695    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2mpq"]
Sep 10 17:13:53.305846 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.305805    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:53.305947 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:53.305929    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:53.306286 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.306257    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7w82"]
Sep 10 17:13:53.306375 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.306359    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:53.306478 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:53.306458    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:53.427246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.427137    2554 generic.go:356] "Generic (PLEG): container finished" podID="873150c3-9650-4c7e-b4ba-5f44b32f2127" containerID="a1ec23f477e8cf4041619ea5059633b79fb7958cd902e0e158aadbdaf4c29ab5" exitCode=0
Sep 10 17:13:53.427246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:53.427234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerDied","Data":"a1ec23f477e8cf4041619ea5059633b79fb7958cd902e0e158aadbdaf4c29ab5"}
Sep 10 17:13:54.432150 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:54.432063    2554 generic.go:356] "Generic (PLEG): container finished" podID="873150c3-9650-4c7e-b4ba-5f44b32f2127" containerID="970c6434362669876e35872bd4b21cfe5a105261c0536892eaacf2054d6be90d" exitCode=0
Sep 10 17:13:54.432549 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:54.432147    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerDied","Data":"970c6434362669876e35872bd4b21cfe5a105261c0536892eaacf2054d6be90d"}
Sep 10 17:13:55.110226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:55.110193    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:55.110436 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:55.110300    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:55.110436 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:55.110346    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:55.110436 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:55.110335    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:55.110699 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:55.110429    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:55.110699 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:55.110510    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:57.109101 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:57.109057    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:57.109552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:57.109058    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:57.109552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:57.109057    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:57.109552 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.109229    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q2mpq" podUID="00ad883b-3f74-42a9-957a-c6369c41581b"
Sep 10 17:13:57.109552 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.109322    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7w82" podUID="d1cbbfd7-c7ab-4420-8376-3cbd71ed8829"
Sep 10 17:13:57.109552 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.109409    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-499fh" podUID="4ff0d9fd-0612-4d96-b385-235ce5bcbed0"
Sep 10 17:13:57.827278 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:57.827232    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:57.827472 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:57.827291    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:57.827472 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.827405    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:57.827472 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.827404    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:57.827676 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.827464    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret podName:4ff0d9fd-0612-4d96-b385-235ce5bcbed0 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:29.827447389 +0000 UTC m=+65.183937782 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret") pod "global-pull-secret-syncer-499fh" (UID: "4ff0d9fd-0612-4d96-b385-235ce5bcbed0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:13:57.827676 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.827599    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:29.827573469 +0000 UTC m=+65.184063860 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:13:57.927980 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:57.927940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:57.928156 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.928112    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:13:57.928156 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.928137    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:13:57.928156 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.928152    2554 projected.go:196] Error preparing data for projected volume kube-api-access-97xrt for pod openshift-network-diagnostics/network-check-target-q2mpq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:57.928290 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:57.928227    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt podName:00ad883b-3f74-42a9-957a-c6369c41581b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:29.928209434 +0000 UTC m=+65.284699819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xrt" (UniqueName: "kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt") pod "network-check-target-q2mpq" (UID: "00ad883b-3f74-42a9-957a-c6369c41581b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:13:58.945806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:58.945774    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-52.ec2.internal" event="NodeReady"
Sep 10 17:13:58.946229 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:58.945952    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 17:13:58.991776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:58.991745    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8m29g"]
Sep 10 17:13:59.017051 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.017029    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7mrft"]
Sep 10 17:13:59.017165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.017144    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.020461 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.020441    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 17:13:59.020599 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.020480    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 17:13:59.020692 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.020680    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2rm4x\""
Sep 10 17:13:59.033743 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.033725    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8m29g"]
Sep 10 17:13:59.034227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.033773    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7mrft"]
Sep 10 17:13:59.034227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.033856    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.036841 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.036820    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 17:13:59.037073 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.037056    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 17:13:59.037147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.037092    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 17:13:59.037147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.037092    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nn99k\""
Sep 10 17:13:59.108392 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.108331    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:13:59.108392 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.108331    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:13:59.108613 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.108343    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:13:59.111302 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.111282    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:13:59.111302 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.111295    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:13:59.111473 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.111380    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7xkh\""
Sep 10 17:13:59.111473 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.111444    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n4rqr\""
Sep 10 17:13:59.111673 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.111503    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:13:59.111749 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.111734    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:13:59.137009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.136990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.137117 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.137043    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/098a0f94-e1af-4846-b58b-d4e132ae8d7b-config-volume\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.137117 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.137094    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098a0f94-e1af-4846-b58b-d4e132ae8d7b-tmp-dir\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.137226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.137118    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b85b6068-2505-4ed6-9dfa-7e5f9841a3a9-cert\") pod \"ingress-canary-7mrft\" (UID: \"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9\") " pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.137226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.137180    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22v9q\" (UniqueName: \"kubernetes.io/projected/b85b6068-2505-4ed6-9dfa-7e5f9841a3a9-kube-api-access-22v9q\") pod \"ingress-canary-7mrft\" (UID: \"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9\") " pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.137226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.137208    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsfn\" (UniqueName: \"kubernetes.io/projected/098a0f94-e1af-4846-b58b-d4e132ae8d7b-kube-api-access-2vsfn\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.238316 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238281    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/098a0f94-e1af-4846-b58b-d4e132ae8d7b-config-volume\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.238451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238324    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098a0f94-e1af-4846-b58b-d4e132ae8d7b-tmp-dir\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.238451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238343    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b85b6068-2505-4ed6-9dfa-7e5f9841a3a9-cert\") pod \"ingress-canary-7mrft\" (UID: \"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9\") " pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.238451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22v9q\" (UniqueName: \"kubernetes.io/projected/b85b6068-2505-4ed6-9dfa-7e5f9841a3a9-kube-api-access-22v9q\") pod \"ingress-canary-7mrft\" (UID: \"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9\") " pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.238451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238418    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsfn\" (UniqueName: \"kubernetes.io/projected/098a0f94-e1af-4846-b58b-d4e132ae8d7b-kube-api-access-2vsfn\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.238672 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238475    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.238672 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:59.238612    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:13:59.238768 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:59.238677    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls podName:098a0f94-e1af-4846-b58b-d4e132ae8d7b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:59.738660444 +0000 UTC m=+35.095150822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls") pod "dns-default-8m29g" (UID: "098a0f94-e1af-4846-b58b-d4e132ae8d7b") : secret "dns-default-metrics-tls" not found
Sep 10 17:13:59.238824 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238758    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098a0f94-e1af-4846-b58b-d4e132ae8d7b-tmp-dir\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.238982 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.238953    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/098a0f94-e1af-4846-b58b-d4e132ae8d7b-config-volume\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.242719 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.242605    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b85b6068-2505-4ed6-9dfa-7e5f9841a3a9-cert\") pod \"ingress-canary-7mrft\" (UID: \"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9\") " pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.247298 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.247276    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsfn\" (UniqueName: \"kubernetes.io/projected/098a0f94-e1af-4846-b58b-d4e132ae8d7b-kube-api-access-2vsfn\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.247678 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.247659    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22v9q\" (UniqueName: \"kubernetes.io/projected/b85b6068-2505-4ed6-9dfa-7e5f9841a3a9-kube-api-access-22v9q\") pod \"ingress-canary-7mrft\" (UID: \"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9\") " pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.342736 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.342708    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7mrft"
Sep 10 17:13:59.457887 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.457848    2554 generic.go:356] "Generic (PLEG): container finished" podID="873150c3-9650-4c7e-b4ba-5f44b32f2127" containerID="ebb1dc338e3ffd6ea5044e35112bde6ed7c0799b0c179ec130e1a74ca6759460" exitCode=0
Sep 10 17:13:59.458048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.457927    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerDied","Data":"ebb1dc338e3ffd6ea5044e35112bde6ed7c0799b0c179ec130e1a74ca6759460"}
Sep 10 17:13:59.664976 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.664945    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7mrft"]
Sep 10 17:13:59.681378 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:13:59.681349    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85b6068_2505_4ed6_9dfa_7e5f9841a3a9.slice/crio-4f04b1b7cfca337a2faa79de18ce4a5e1cf989f9d0bff284a2067406c65c928d WatchSource:0}: Error finding container 4f04b1b7cfca337a2faa79de18ce4a5e1cf989f9d0bff284a2067406c65c928d: Status 404 returned error can't find the container with id 4f04b1b7cfca337a2faa79de18ce4a5e1cf989f9d0bff284a2067406c65c928d
Sep 10 17:13:59.742144 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:13:59.742108    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:13:59.742312 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:59.742249    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:13:59.742312 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:13:59.742302    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls podName:098a0f94-e1af-4846-b58b-d4e132ae8d7b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:00.742288719 +0000 UTC m=+36.098779102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls") pod "dns-default-8m29g" (UID: "098a0f94-e1af-4846-b58b-d4e132ae8d7b") : secret "dns-default-metrics-tls" not found
Sep 10 17:14:00.461834 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:00.461789    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7mrft" event={"ID":"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9","Type":"ContainerStarted","Data":"4f04b1b7cfca337a2faa79de18ce4a5e1cf989f9d0bff284a2067406c65c928d"}
Sep 10 17:14:00.465514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:00.465482    2554 generic.go:356] "Generic (PLEG): container finished" podID="873150c3-9650-4c7e-b4ba-5f44b32f2127" containerID="39e5a1de784dce01051391bd6755ed8dc64921c25cc55bdc9981f495587f05e8" exitCode=0
Sep 10 17:14:00.465656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:00.465550    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerDied","Data":"39e5a1de784dce01051391bd6755ed8dc64921c25cc55bdc9981f495587f05e8"}
Sep 10 17:14:00.750487 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:00.750396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:14:00.750668 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:00.750540    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:14:00.750668 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:00.750611    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls podName:098a0f94-e1af-4846-b58b-d4e132ae8d7b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:02.750595718 +0000 UTC m=+38.107086097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls") pod "dns-default-8m29g" (UID: "098a0f94-e1af-4846-b58b-d4e132ae8d7b") : secret "dns-default-metrics-tls" not found
Sep 10 17:14:01.471967 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:01.471930    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" event={"ID":"873150c3-9650-4c7e-b4ba-5f44b32f2127","Type":"ContainerStarted","Data":"9496d92cb1b97d4fcc9289c74e70a15f73d0e748e32f7968cb4857b805874fae"}
Sep 10 17:14:02.475754 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:02.475501    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7mrft" event={"ID":"b85b6068-2505-4ed6-9dfa-7e5f9841a3a9","Type":"ContainerStarted","Data":"4bad27969e00dad1c37cc0d9e551a032845d0a89e190900e101821d409f3c52a"}
Sep 10 17:14:02.491575 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:02.491506    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dvwnv" podStartSLOduration=7.399620459 podStartE2EDuration="36.491490258s" totalImagesPullingTime="29.091869799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:01.495113279 +0000 UTC m=+36.851603672" watchObservedRunningTime="2026-09-10 17:14:02.491490258 +0000 UTC m=+37.847980658"
Sep 10 17:14:02.491763 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:02.491743    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7mrft" podStartSLOduration=2.0960311 podStartE2EDuration="4.491736022s" totalImagesPullingTime="2.395704922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:02.491063052 +0000 UTC m=+37.847553451" watchObservedRunningTime="2026-09-10 17:14:02.491736022 +0000 UTC m=+37.848226420"
Sep 10 17:14:02.766710 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:02.766629    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:14:02.766843 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:02.766788    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:14:02.766881 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:02.766853    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls podName:098a0f94-e1af-4846-b58b-d4e132ae8d7b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:06.766835024 +0000 UTC m=+42.123325404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls") pod "dns-default-8m29g" (UID: "098a0f94-e1af-4846-b58b-d4e132ae8d7b") : secret "dns-default-metrics-tls" not found
Sep 10 17:14:06.793925 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:06.793886    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:14:06.794329 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:06.794048    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:14:06.794329 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:06.794116    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls podName:098a0f94-e1af-4846-b58b-d4e132ae8d7b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:14.794100594 +0000 UTC m=+50.150590996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls") pod "dns-default-8m29g" (UID: "098a0f94-e1af-4846-b58b-d4e132ae8d7b") : secret "dns-default-metrics-tls" not found
Sep 10 17:14:14.850842 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:14.850799    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:14:14.851235 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:14.850948    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:14:14.851235 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:14.851019    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls podName:098a0f94-e1af-4846-b58b-d4e132ae8d7b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:30.851002544 +0000 UTC m=+66.207492922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls") pod "dns-default-8m29g" (UID: "098a0f94-e1af-4846-b58b-d4e132ae8d7b") : secret "dns-default-metrics-tls" not found
Sep 10 17:14:24.442698 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:24.442672    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw9fz"
Sep 10 17:14:29.850320 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.850276    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:14:29.850807 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.850335    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:14:29.852864 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.852839    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:14:29.852864 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.852856    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:14:29.861416 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:29.861392    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 17:14:29.861495 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:29.861454    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs podName:d1cbbfd7-c7ab-4420-8376-3cbd71ed8829 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:33.861438041 +0000 UTC m=+129.217928433 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs") pod "network-metrics-daemon-n7w82" (UID: "d1cbbfd7-c7ab-4420-8376-3cbd71ed8829") : secret "metrics-daemon-secret" not found
Sep 10 17:14:29.864525 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.864502    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ff0d9fd-0612-4d96-b385-235ce5bcbed0-original-pull-secret\") pod \"global-pull-secret-syncer-499fh\" (UID: \"4ff0d9fd-0612-4d96-b385-235ce5bcbed0\") " pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:14:29.951105 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.951065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:14:29.954523 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.954500    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:14:29.964981 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.964954    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:14:29.975114 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:29.975090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xrt\" (UniqueName: \"kubernetes.io/projected/00ad883b-3f74-42a9-957a-c6369c41581b-kube-api-access-97xrt\") pod \"network-check-target-q2mpq\" (UID: \"00ad883b-3f74-42a9-957a-c6369c41581b\") " pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:14:30.023225 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.023194    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7xkh\""
Sep 10 17:14:30.026385 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.026367    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-499fh"
Sep 10 17:14:30.031650 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.031610    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:14:30.324631 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.324608    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-499fh"]
Sep 10 17:14:30.355857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.355786    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q2mpq"]
Sep 10 17:14:30.358587 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:14:30.358564    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ad883b_3f74_42a9_957a_c6369c41581b.slice/crio-221b4db0f8ce9b1a0cba8730a5e3f7471a295aa54dbc2227e967ae6f6dc73706 WatchSource:0}: Error finding container 221b4db0f8ce9b1a0cba8730a5e3f7471a295aa54dbc2227e967ae6f6dc73706: Status 404 returned error can't find the container with id 221b4db0f8ce9b1a0cba8730a5e3f7471a295aa54dbc2227e967ae6f6dc73706
Sep 10 17:14:30.540600 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.540554    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2mpq" event={"ID":"00ad883b-3f74-42a9-957a-c6369c41581b","Type":"ContainerStarted","Data":"221b4db0f8ce9b1a0cba8730a5e3f7471a295aa54dbc2227e967ae6f6dc73706"}
Sep 10 17:14:30.541732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.541705    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-499fh" event={"ID":"4ff0d9fd-0612-4d96-b385-235ce5bcbed0","Type":"ContainerStarted","Data":"88376818edec3f28cc2101902b45af3248f19409c8f90a2e77fd49e03d1c689c"}
Sep 10 17:14:30.856790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:30.856755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:14:30.857263 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:30.856913    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:14:30.857263 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:30.856992    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls podName:098a0f94-e1af-4846-b58b-d4e132ae8d7b nodeName:}" failed. No retries permitted until 2026-09-10 17:15:02.856973785 +0000 UTC m=+98.213464178 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls") pod "dns-default-8m29g" (UID: "098a0f94-e1af-4846-b58b-d4e132ae8d7b") : secret "dns-default-metrics-tls" not found
Sep 10 17:14:35.556411 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:35.556371    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q2mpq" event={"ID":"00ad883b-3f74-42a9-957a-c6369c41581b","Type":"ContainerStarted","Data":"4d6e67ce48b4ae51776a15fe7c1791851431d3626cdbb7ce858dfb008888c1ec"}
Sep 10 17:14:35.557006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:35.556511    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:14:35.558045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:35.558024    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-499fh" event={"ID":"4ff0d9fd-0612-4d96-b385-235ce5bcbed0","Type":"ContainerStarted","Data":"8c73d5541de6f44450d117dbcd93f068f3b60bc53fcda1744dcc0cf413c63062"}
Sep 10 17:14:35.593688 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:35.593643    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-499fh" podStartSLOduration=65.062951204 podStartE2EDuration="1m9.593628387s" totalImagesPullingTime="4.530677183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:35.592724154 +0000 UTC m=+70.949214555" watchObservedRunningTime="2026-09-10 17:14:35.593628387 +0000 UTC m=+70.950118786"
Sep 10 17:14:35.593869 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:35.593843    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q2mpq" podStartSLOduration=65.101743605 podStartE2EDuration="1m9.593835665s" totalImagesPullingTime="4.49209206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:35.576710398 +0000 UTC m=+70.933200798" watchObservedRunningTime="2026-09-10 17:14:35.593835665 +0000 UTC m=+70.950326067"
Sep 10 17:14:40.973011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:40.972972    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"]
Sep 10 17:14:40.974878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:40.974861    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"
Sep 10 17:14:40.977552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:40.977515    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:14:40.978453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:40.978434    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nngnp\""
Sep 10 17:14:40.978556 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:40.978458    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 17:14:40.985544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:40.985512    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"]
Sep 10 17:14:41.021118 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.021084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9v7r\" (UniqueName: \"kubernetes.io/projected/6f89f9bc-5619-49ae-9f52-3409196a37f5-kube-api-access-d9v7r\") pod \"volume-data-source-validator-695c895bd6-xg8cj\" (UID: \"6f89f9bc-5619-49ae-9f52-3409196a37f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"
Sep 10 17:14:41.062818 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.062790    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"]
Sep 10 17:14:41.064490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.064476    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.067037 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.067015    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:14:41.067037 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.067022    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 17:14:41.067185 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.067020    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 17:14:41.067185 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.067172    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ghxw5\""
Sep 10 17:14:41.067312 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.067298    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 17:14:41.077902 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.077883    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"]
Sep 10 17:14:41.079453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.079432    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z7qwj"]
Sep 10 17:14:41.079572 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.079544    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:41.081197 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.081173    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4dk2c"]
Sep 10 17:14:41.081308 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.081294    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.082442 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.082420    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-m24ph\""
Sep 10 17:14:41.082718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.082701    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 17:14:41.082878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.082857    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"]
Sep 10 17:14:41.082950 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.082925    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:14:41.082997 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.082956    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 17:14:41.083045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.082962    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.083694 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.083676    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 17:14:41.083782 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.083733    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:14:41.083843 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.083788    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7gn9t\""
Sep 10 17:14:41.084206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.084191    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 17:14:41.084796 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.084779    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 17:14:41.086679 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.086656    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 17:14:41.086774 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.086733    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 17:14:41.086774 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.086761    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-br9dn\""
Sep 10 17:14:41.086880 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.086782    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 17:14:41.086880 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.086665    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 17:14:41.096702 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.096676    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 17:14:41.097093 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.097071    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 17:14:41.097566 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.097498    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4dk2c"]
Sep 10 17:14:41.099938 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.099919    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z7qwj"]
Sep 10 17:14:41.100820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.100799    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"]
Sep 10 17:14:41.122049 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122024    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b786b74c-bc04-4817-b490-c2d209a18cdb-config\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.122049 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122050    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b786b74c-bc04-4817-b490-c2d209a18cdb-trusted-ca\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.122239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122071    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b30b0823-6551-4431-91ef-bccae87c4810-snapshots\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.122239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122091    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:41.122239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122106    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b30b0823-6551-4431-91ef-bccae87c4810-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.122239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122123    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b30b0823-6551-4431-91ef-bccae87c4810-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.122239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122155    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-serving-cert\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.122239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122176    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7q2x\" (UniqueName: \"kubernetes.io/projected/b30b0823-6551-4431-91ef-bccae87c4810-kube-api-access-h7q2x\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.122239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b786b74c-bc04-4817-b490-c2d209a18cdb-serving-cert\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.122460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122302    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b30b0823-6551-4431-91ef-bccae87c4810-serving-cert\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.122460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5n2t\" (UniqueName: \"kubernetes.io/projected/b786b74c-bc04-4817-b490-c2d209a18cdb-kube-api-access-g5n2t\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.122460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122372    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b30b0823-6551-4431-91ef-bccae87c4810-tmp\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.122460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122391    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hlwd\" (UniqueName: \"kubernetes.io/projected/b9687b3c-7d88-476b-a6a8-c73f728294a0-kube-api-access-6hlwd\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:41.122460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122431    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9cw\" (UniqueName: \"kubernetes.io/projected/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-kube-api-access-jf9cw\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.122460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122455    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9v7r\" (UniqueName: \"kubernetes.io/projected/6f89f9bc-5619-49ae-9f52-3409196a37f5-kube-api-access-d9v7r\") pod \"volume-data-source-validator-695c895bd6-xg8cj\" (UID: \"6f89f9bc-5619-49ae-9f52-3409196a37f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"
Sep 10 17:14:41.122681 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.122469    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-config\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.133329 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.133298    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9v7r\" (UniqueName: \"kubernetes.io/projected/6f89f9bc-5619-49ae-9f52-3409196a37f5-kube-api-access-d9v7r\") pod \"volume-data-source-validator-695c895bd6-xg8cj\" (UID: \"6f89f9bc-5619-49ae-9f52-3409196a37f5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"
Sep 10 17:14:41.223648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223548    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hlwd\" (UniqueName: \"kubernetes.io/projected/b9687b3c-7d88-476b-a6a8-c73f728294a0-kube-api-access-6hlwd\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:41.223648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223632    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9cw\" (UniqueName: \"kubernetes.io/projected/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-kube-api-access-jf9cw\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.223863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-config\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.223863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223688    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b786b74c-bc04-4817-b490-c2d209a18cdb-config\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.223863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223786    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b786b74c-bc04-4817-b490-c2d209a18cdb-trusted-ca\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.223863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223815    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b30b0823-6551-4431-91ef-bccae87c4810-snapshots\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.223863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223845    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:41.224106 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223869    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b30b0823-6551-4431-91ef-bccae87c4810-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224106 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223893    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b30b0823-6551-4431-91ef-bccae87c4810-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224106 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-serving-cert\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.224106 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7q2x\" (UniqueName: \"kubernetes.io/projected/b30b0823-6551-4431-91ef-bccae87c4810-kube-api-access-h7q2x\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224106 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:41.223972    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:14:41.224106 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:41.224054    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls podName:b9687b3c-7d88-476b-a6a8-c73f728294a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:41.72403373 +0000 UTC m=+77.080524125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-x4gdm" (UID: "b9687b3c-7d88-476b-a6a8-c73f728294a0") : secret "samples-operator-tls" not found
Sep 10 17:14:41.224106 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.223982    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b786b74c-bc04-4817-b490-c2d209a18cdb-serving-cert\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.224452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224122    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b30b0823-6551-4431-91ef-bccae87c4810-serving-cert\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224160    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5n2t\" (UniqueName: \"kubernetes.io/projected/b786b74c-bc04-4817-b490-c2d209a18cdb-kube-api-access-g5n2t\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.224452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224200    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b30b0823-6551-4431-91ef-bccae87c4810-tmp\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224336    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-config\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.224737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224452    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b786b74c-bc04-4817-b490-c2d209a18cdb-config\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.224737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224470    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b30b0823-6551-4431-91ef-bccae87c4810-snapshots\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224492    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b30b0823-6551-4431-91ef-bccae87c4810-tmp\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224689    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b30b0823-6551-4431-91ef-bccae87c4810-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.224981 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.224961    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b30b0823-6551-4431-91ef-bccae87c4810-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.225309 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.225289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b786b74c-bc04-4817-b490-c2d209a18cdb-trusted-ca\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.226655 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.226631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-serving-cert\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.226940 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.226924    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b786b74c-bc04-4817-b490-c2d209a18cdb-serving-cert\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.226984 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.226967    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b30b0823-6551-4431-91ef-bccae87c4810-serving-cert\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.244956 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.244927    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hlwd\" (UniqueName: \"kubernetes.io/projected/b9687b3c-7d88-476b-a6a8-c73f728294a0-kube-api-access-6hlwd\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:41.246247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.246222    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9cw\" (UniqueName: \"kubernetes.io/projected/eb3ae1d4-2da4-4635-93cf-51fa19c46dc8-kube-api-access-jf9cw\") pod \"service-ca-operator-859f94c4b-rrnqn\" (UID: \"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.247139 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.247115    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5n2t\" (UniqueName: \"kubernetes.io/projected/b786b74c-bc04-4817-b490-c2d209a18cdb-kube-api-access-g5n2t\") pod \"console-operator-85b454874b-z7qwj\" (UID: \"b786b74c-bc04-4817-b490-c2d209a18cdb\") " pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.248432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.248411    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7q2x\" (UniqueName: \"kubernetes.io/projected/b30b0823-6551-4431-91ef-bccae87c4810-kube-api-access-h7q2x\") pod \"insights-operator-76bcd6ccf5-4dk2c\" (UID: \"b30b0823-6551-4431-91ef-bccae87c4810\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.284109 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.284084    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"
Sep 10 17:14:41.372945 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.372911    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"
Sep 10 17:14:41.399620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.399587    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:41.405391 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.405366    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c"
Sep 10 17:14:41.427165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.427135    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77bcd77446-rs27j"]
Sep 10 17:14:41.438967 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.438938    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.442463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.442139    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 17:14:41.442463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.442184    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 17:14:41.442463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.442315    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gj2cb\""
Sep 10 17:14:41.442463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.442400    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 17:14:41.445274 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.445248    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bcd77446-rs27j"]
Sep 10 17:14:41.447509 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.447368    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 17:14:41.526288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526253    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-certificates\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.526423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526291    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-bound-sa-token\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.526423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526320    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-trusted-ca\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.526423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526399    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-ca-trust-extracted\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.526524 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526450    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-installation-pull-secrets\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.526524 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526491    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpz6\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-kube-api-access-rnpz6\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.526602 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.526602 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.526585    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-image-registry-private-configuration\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.550009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.549977    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj"]
Sep 10 17:14:41.554771 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:14:41.554749    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f89f9bc_5619_49ae_9f52_3409196a37f5.slice/crio-9af47c95849d057a1ce1c5e00faad401e0a720f704dd43eeb723eef431b3ec13 WatchSource:0}: Error finding container 9af47c95849d057a1ce1c5e00faad401e0a720f704dd43eeb723eef431b3ec13: Status 404 returned error can't find the container with id 9af47c95849d057a1ce1c5e00faad401e0a720f704dd43eeb723eef431b3ec13
Sep 10 17:14:41.573234 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.573199    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj" event={"ID":"6f89f9bc-5619-49ae-9f52-3409196a37f5","Type":"ContainerStarted","Data":"9af47c95849d057a1ce1c5e00faad401e0a720f704dd43eeb723eef431b3ec13"}
Sep 10 17:14:41.627722 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.627684    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-installation-pull-secrets\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.627857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.627727    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpz6\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-kube-api-access-rnpz6\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.627857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.627755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.627857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.627787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-image-registry-private-configuration\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.627857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.627835    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-certificates\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.628050 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:41.627862    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:14:41.628050 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:41.627881    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bcd77446-rs27j: secret "image-registry-tls" not found
Sep 10 17:14:41.628050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.627917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-bound-sa-token\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.628050 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:41.627945    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls podName:e1c5d94e-d9ff-454c-a813-3a5a78e1c227 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:42.127924908 +0000 UTC m=+77.484415291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls") pod "image-registry-77bcd77446-rs27j" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227") : secret "image-registry-tls" not found
Sep 10 17:14:41.628050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.627974    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-trusted-ca\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.628050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.628036    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-ca-trust-extracted\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.628459 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.628436    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-ca-trust-extracted\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.628517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.628463    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-certificates\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.628842 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.628822    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-trusted-ca\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.630169 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.630150    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-image-registry-private-configuration\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.630225 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.630199    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-installation-pull-secrets\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.635641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.635614    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-bound-sa-token\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.635720 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.635647    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpz6\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-kube-api-access-rnpz6\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:41.641075 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.641055    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn"]
Sep 10 17:14:41.643262 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:14:41.643241    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3ae1d4_2da4_4635_93cf_51fa19c46dc8.slice/crio-a17f7fe107769a59bf7845f8b8bb00383f41c8f14bd7fab23575833a6080e123 WatchSource:0}: Error finding container a17f7fe107769a59bf7845f8b8bb00383f41c8f14bd7fab23575833a6080e123: Status 404 returned error can't find the container with id a17f7fe107769a59bf7845f8b8bb00383f41c8f14bd7fab23575833a6080e123
Sep 10 17:14:41.676156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.676126    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z7qwj"]
Sep 10 17:14:41.680039 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:14:41.679999    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb786b74c_bc04_4817_b490_c2d209a18cdb.slice/crio-b90f4d6edc7f720976546b81de97deb950ad11b6a14eace6456d466992934c19 WatchSource:0}: Error finding container b90f4d6edc7f720976546b81de97deb950ad11b6a14eace6456d466992934c19: Status 404 returned error can't find the container with id b90f4d6edc7f720976546b81de97deb950ad11b6a14eace6456d466992934c19
Sep 10 17:14:41.699416 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.699381    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4dk2c"]
Sep 10 17:14:41.702180 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:14:41.702144    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30b0823_6551_4431_91ef_bccae87c4810.slice/crio-223fd92d3a51239c775879e602cbfd56372a20df5850084fafd99293aa62b2d1 WatchSource:0}: Error finding container 223fd92d3a51239c775879e602cbfd56372a20df5850084fafd99293aa62b2d1: Status 404 returned error can't find the container with id 223fd92d3a51239c775879e602cbfd56372a20df5850084fafd99293aa62b2d1
Sep 10 17:14:41.728995 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:41.728892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:41.729141 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:41.729051    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:14:41.729141 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:41.729137    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls podName:b9687b3c-7d88-476b-a6a8-c73f728294a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:42.729120567 +0000 UTC m=+78.085610946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-x4gdm" (UID: "b9687b3c-7d88-476b-a6a8-c73f728294a0") : secret "samples-operator-tls" not found
Sep 10 17:14:42.132015 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:42.131970    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:42.132489 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:42.132164    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:14:42.132489 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:42.132185    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bcd77446-rs27j: secret "image-registry-tls" not found
Sep 10 17:14:42.132489 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:42.132245    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls podName:e1c5d94e-d9ff-454c-a813-3a5a78e1c227 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:43.132225792 +0000 UTC m=+78.488716184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls") pod "image-registry-77bcd77446-rs27j" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227") : secret "image-registry-tls" not found
Sep 10 17:14:42.578750 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:42.578663    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn" event={"ID":"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8","Type":"ContainerStarted","Data":"a17f7fe107769a59bf7845f8b8bb00383f41c8f14bd7fab23575833a6080e123"}
Sep 10 17:14:42.580258 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:42.580190    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c" event={"ID":"b30b0823-6551-4431-91ef-bccae87c4810","Type":"ContainerStarted","Data":"223fd92d3a51239c775879e602cbfd56372a20df5850084fafd99293aa62b2d1"}
Sep 10 17:14:42.581785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:42.581761    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" event={"ID":"b786b74c-bc04-4817-b490-c2d209a18cdb","Type":"ContainerStarted","Data":"b90f4d6edc7f720976546b81de97deb950ad11b6a14eace6456d466992934c19"}
Sep 10 17:14:42.738542 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:42.738254    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:42.738542 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:42.738428    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:14:42.738542 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:42.738520    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls podName:b9687b3c-7d88-476b-a6a8-c73f728294a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:44.738498381 +0000 UTC m=+80.094988760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-x4gdm" (UID: "b9687b3c-7d88-476b-a6a8-c73f728294a0") : secret "samples-operator-tls" not found
Sep 10 17:14:43.142146 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:43.142108    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:43.142516 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:43.142257    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:14:43.142516 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:43.142272    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bcd77446-rs27j: secret "image-registry-tls" not found
Sep 10 17:14:43.142516 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:43.142330    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls podName:e1c5d94e-d9ff-454c-a813-3a5a78e1c227 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:45.142309824 +0000 UTC m=+80.498800265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls") pod "image-registry-77bcd77446-rs27j" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227") : secret "image-registry-tls" not found
Sep 10 17:14:43.586601 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:43.586558    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj" event={"ID":"6f89f9bc-5619-49ae-9f52-3409196a37f5","Type":"ContainerStarted","Data":"544d8748c60100ac80684d0220b7b0920e01ec1e6a03790d37a23e002e389b68"}
Sep 10 17:14:43.604010 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:43.603954    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xg8cj" podStartSLOduration=2.478303915 podStartE2EDuration="3.603935096s" totalImagesPullingTime="1.125631181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:43.60233355 +0000 UTC m=+78.958823950" watchObservedRunningTime="2026-09-10 17:14:43.603935096 +0000 UTC m=+78.960425497"
Sep 10 17:14:44.755866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:44.755762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:44.756198 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:44.755915    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:14:44.756198 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:44.756030    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls podName:b9687b3c-7d88-476b-a6a8-c73f728294a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:48.756006097 +0000 UTC m=+84.112496489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-x4gdm" (UID: "b9687b3c-7d88-476b-a6a8-c73f728294a0") : secret "samples-operator-tls" not found
Sep 10 17:14:45.158587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.158556    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:45.158762 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:45.158714    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:14:45.158762 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:45.158734    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bcd77446-rs27j: secret "image-registry-tls" not found
Sep 10 17:14:45.158855 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:45.158797    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls podName:e1c5d94e-d9ff-454c-a813-3a5a78e1c227 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:49.158777433 +0000 UTC m=+84.515267814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls") pod "image-registry-77bcd77446-rs27j" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227") : secret "image-registry-tls" not found
Sep 10 17:14:45.594262 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.594224    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c" event={"ID":"b30b0823-6551-4431-91ef-bccae87c4810","Type":"ContainerStarted","Data":"40b8176192111ed4922cf40849abb04fc52664df7fb23dd14fca0be7c4ab274b"}
Sep 10 17:14:45.595941 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.595924    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/0.log"
Sep 10 17:14:45.596028 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.595957    2554 generic.go:356] "Generic (PLEG): container finished" podID="b786b74c-bc04-4817-b490-c2d209a18cdb" containerID="42fc7091f7a006493f3420cf872d78ad7630f54efe99b54b93178bcf17711048" exitCode=255
Sep 10 17:14:45.596077 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.596029    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" event={"ID":"b786b74c-bc04-4817-b490-c2d209a18cdb","Type":"ContainerDied","Data":"42fc7091f7a006493f3420cf872d78ad7630f54efe99b54b93178bcf17711048"}
Sep 10 17:14:45.596235 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.596213    2554 scope.go:117] "RemoveContainer" containerID="42fc7091f7a006493f3420cf872d78ad7630f54efe99b54b93178bcf17711048"
Sep 10 17:14:45.597838 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.597813    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn" event={"ID":"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8","Type":"ContainerStarted","Data":"38e33a6d5203f5fbd9fd05efca6ca1894756624926bf00f767a29d4a249f63a6"}
Sep 10 17:14:45.614777 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.614725    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c" podStartSLOduration=1.626229951 podStartE2EDuration="4.614707388s" totalImagesPullingTime="2.988477437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:45.61328826 +0000 UTC m=+80.969778661" watchObservedRunningTime="2026-09-10 17:14:45.614707388 +0000 UTC m=+80.971197789"
Sep 10 17:14:45.635917 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.635876    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn" podStartSLOduration=1.581574206 podStartE2EDuration="4.635861219s" totalImagesPullingTime="3.054287013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:45.635163125 +0000 UTC m=+80.991653525" watchObservedRunningTime="2026-09-10 17:14:45.635861219 +0000 UTC m=+80.992351618"
Sep 10 17:14:45.996904 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.996826    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cz8pq"]
Sep 10 17:14:45.998910 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:45.998893    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.001766 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.001744    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 17:14:46.002516 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.002491    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 17:14:46.002853 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.002835    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2hvq7\""
Sep 10 17:14:46.014437 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.014414    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cz8pq"]
Sep 10 17:14:46.066358 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.066327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.066496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.066374    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-data-volume\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.066496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.066406    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.066496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.066434    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pld46\" (UniqueName: \"kubernetes.io/projected/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-api-access-pld46\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.066496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.066453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-crio-socket\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.167249 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.167212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.167389 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.167261    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-data-volume\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.167389 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.167283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.167389 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.167304    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pld46\" (UniqueName: \"kubernetes.io/projected/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-api-access-pld46\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.167389 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.167321    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-crio-socket\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.167389 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.167341    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:14:46.167573 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.167397    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:46.167573 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.167415    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:46.667393721 +0000 UTC m=+82.023884115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : configmap "kube-rbac-proxy" not found
Sep 10 17:14:46.167573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.167400    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-crio-socket\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.167573 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.167459    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:46.667441522 +0000 UTC m=+82.023931917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:46.167731 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.167596    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-data-volume\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.180811 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.180784    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pld46\" (UniqueName: \"kubernetes.io/projected/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-api-access-pld46\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.602301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.602272    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/1.log"
Sep 10 17:14:46.602697 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.602683    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/0.log"
Sep 10 17:14:46.602780 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.602717    2554 generic.go:356] "Generic (PLEG): container finished" podID="b786b74c-bc04-4817-b490-c2d209a18cdb" containerID="786ed6dff2196304e389c3f7ee0a00b34fb3d63e92ff01016f68fb3fedb821aa" exitCode=255
Sep 10 17:14:46.602829 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.602801    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" event={"ID":"b786b74c-bc04-4817-b490-c2d209a18cdb","Type":"ContainerDied","Data":"786ed6dff2196304e389c3f7ee0a00b34fb3d63e92ff01016f68fb3fedb821aa"}
Sep 10 17:14:46.602876 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.602862    2554 scope.go:117] "RemoveContainer" containerID="42fc7091f7a006493f3420cf872d78ad7630f54efe99b54b93178bcf17711048"
Sep 10 17:14:46.603047 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.603028    2554 scope.go:117] "RemoveContainer" containerID="786ed6dff2196304e389c3f7ee0a00b34fb3d63e92ff01016f68fb3fedb821aa"
Sep 10 17:14:46.603227 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.603204    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z7qwj_openshift-console-operator(b786b74c-bc04-4817-b490-c2d209a18cdb)\"" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" podUID="b786b74c-bc04-4817-b490-c2d209a18cdb"
Sep 10 17:14:46.672620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.672591    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.672791 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.672775    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:14:46.672849 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.672834    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:47.672815328 +0000 UTC m=+83.029305720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : configmap "kube-rbac-proxy" not found
Sep 10 17:14:46.673048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:46.673029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:46.673116 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.673092    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:46.673171 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:46.673148    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:47.673130677 +0000 UTC m=+83.029621070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:47.607883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:47.607759    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/1.log"
Sep 10 17:14:47.608338 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:47.608158    2554 scope.go:117] "RemoveContainer" containerID="786ed6dff2196304e389c3f7ee0a00b34fb3d63e92ff01016f68fb3fedb821aa"
Sep 10 17:14:47.608401 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:47.608346    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z7qwj_openshift-console-operator(b786b74c-bc04-4817-b490-c2d209a18cdb)\"" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" podUID="b786b74c-bc04-4817-b490-c2d209a18cdb"
Sep 10 17:14:47.680892 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:47.680857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:47.681085 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:47.680951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:47.681085 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:47.681010    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:47.681085 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:47.681086    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:49.68106451 +0000 UTC m=+85.037554902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:47.681227 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:47.681090    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:14:47.681227 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:47.681166    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:49.68115002 +0000 UTC m=+85.037640406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : configmap "kube-rbac-proxy" not found
Sep 10 17:14:48.735779 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:48.735750    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ss6gj_22d0f763-d51f-4444-89fd-0732b083afa3/dns-node-resolver/0.log"
Sep 10 17:14:48.790304 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:48.790260    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:48.790453 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:48.790401    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 17:14:48.790492 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:48.790474    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls podName:b9687b3c-7d88-476b-a6a8-c73f728294a0 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:56.790457828 +0000 UTC m=+92.146948228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-x4gdm" (UID: "b9687b3c-7d88-476b-a6a8-c73f728294a0") : secret "samples-operator-tls" not found
Sep 10 17:14:49.193513 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:49.193477    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:49.193687 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:49.193623    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 17:14:49.193687 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:49.193634    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bcd77446-rs27j: secret "image-registry-tls" not found
Sep 10 17:14:49.193687 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:49.193674    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls podName:e1c5d94e-d9ff-454c-a813-3a5a78e1c227 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:57.193662271 +0000 UTC m=+92.550152649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls") pod "image-registry-77bcd77446-rs27j" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227") : secret "image-registry-tls" not found
Sep 10 17:14:49.696727 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:49.696687    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:49.696907 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:49.696747    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:49.696907 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:49.696814    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:14:49.696907 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:49.696858    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:49.696907 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:49.696893    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:53.69687627 +0000 UTC m=+89.053366648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : configmap "kube-rbac-proxy" not found
Sep 10 17:14:49.696907 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:49.696908    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:14:53.696902426 +0000 UTC m=+89.053392803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:49.935443 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:49.935415    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zpxkk_e24a6e41-b03d-4e48-be14-b7487072c80b/node-ca/0.log"
Sep 10 17:14:50.335829 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:50.335801    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7mrft_b85b6068-2505-4ed6-9dfa-7e5f9841a3a9/serve-healthcheck-canary/0.log"
Sep 10 17:14:51.399837 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:51.399808    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:51.400187 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:51.399856    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:14:51.400187 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:51.400137    2554 scope.go:117] "RemoveContainer" containerID="786ed6dff2196304e389c3f7ee0a00b34fb3d63e92ff01016f68fb3fedb821aa"
Sep 10 17:14:51.400301 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:51.400285    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z7qwj_openshift-console-operator(b786b74c-bc04-4817-b490-c2d209a18cdb)\"" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" podUID="b786b74c-bc04-4817-b490-c2d209a18cdb"
Sep 10 17:14:53.727640 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:53.727593    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:53.728046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:53.727696    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:14:53.728046 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:53.727749    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:53.728046 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:53.727777    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:14:53.728046 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:53.727828    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:15:01.72781102 +0000 UTC m=+97.084301397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:14:53.728046 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:14:53.727843    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:15:01.727836637 +0000 UTC m=+97.084327014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : configmap "kube-rbac-proxy" not found
Sep 10 17:14:56.851334 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:56.851298    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:56.853631 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:56.853596    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9687b3c-7d88-476b-a6a8-c73f728294a0-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-x4gdm\" (UID: \"b9687b3c-7d88-476b-a6a8-c73f728294a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:56.990367 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:56.990329    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"
Sep 10 17:14:57.254189 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:57.254100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:57.255445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:57.255413    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm"]
Sep 10 17:14:57.256771 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:57.256748    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"image-registry-77bcd77446-rs27j\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") " pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:57.353197 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:57.353175    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:57.619018 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:57.618984    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bcd77446-rs27j"]
Sep 10 17:14:57.621438 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:14:57.621416    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c5d94e_d9ff_454c_a813_3a5a78e1c227.slice/crio-5b3a75e4dfbd925edcc338ae4aa32c7850d26dedbda4c49383131545c331917a WatchSource:0}: Error finding container 5b3a75e4dfbd925edcc338ae4aa32c7850d26dedbda4c49383131545c331917a: Status 404 returned error can't find the container with id 5b3a75e4dfbd925edcc338ae4aa32c7850d26dedbda4c49383131545c331917a
Sep 10 17:14:57.639987 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:57.639960    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm" event={"ID":"b9687b3c-7d88-476b-a6a8-c73f728294a0","Type":"ContainerStarted","Data":"cfa2cffbc0efba4d7cf75eeaa12273772cdc0db122f0f25037a884f0f7404601"}
Sep 10 17:14:57.641006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:57.640985    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bcd77446-rs27j" event={"ID":"e1c5d94e-d9ff-454c-a813-3a5a78e1c227","Type":"ContainerStarted","Data":"5b3a75e4dfbd925edcc338ae4aa32c7850d26dedbda4c49383131545c331917a"}
Sep 10 17:14:58.647237 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:58.647192    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bcd77446-rs27j" event={"ID":"e1c5d94e-d9ff-454c-a813-3a5a78e1c227","Type":"ContainerStarted","Data":"5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21"}
Sep 10 17:14:58.647752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:58.647321    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:14:58.669361 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:58.669302    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77bcd77446-rs27j" podStartSLOduration=17.669282101 podStartE2EDuration="17.669282101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:58.668912385 +0000 UTC m=+94.025402785" watchObservedRunningTime="2026-09-10 17:14:58.669282101 +0000 UTC m=+94.025772501"
Sep 10 17:14:59.652453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:59.652402    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm" event={"ID":"b9687b3c-7d88-476b-a6a8-c73f728294a0","Type":"ContainerStarted","Data":"4b78583c8b0abf3b8e75bc4f9908034d83aaf8718cae7e73ac6fb2ce925e01f0"}
Sep 10 17:14:59.652919 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:59.652465    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm" event={"ID":"b9687b3c-7d88-476b-a6a8-c73f728294a0","Type":"ContainerStarted","Data":"4e910f0ce0dbae2ffd1659aa09cbf2a4a1e325a267def70c06a355b6d2ff6066"}
Sep 10 17:14:59.677675 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:14:59.677630    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-x4gdm" podStartSLOduration=17.217501326 podStartE2EDuration="18.677617798s" totalImagesPullingTime="1.460116472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:59.676019912 +0000 UTC m=+95.032510311" watchObservedRunningTime="2026-09-10 17:14:59.677617798 +0000 UTC m=+95.034108197"
Sep 10 17:15:01.794813 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:01.794761    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:15:01.794813 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:01.794827    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:15:01.795238 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:15:01.794914    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:15:01.795238 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:15:01.794964    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls podName:f4f8197d-f3d1-4bc5-8e68-bb82c19b972b nodeName:}" failed. No retries permitted until 2026-09-10 17:15:17.794951308 +0000 UTC m=+113.151441686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz8pq" (UID: "f4f8197d-f3d1-4bc5-8e68-bb82c19b972b") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:15:01.795322 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:01.795302    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:15:02.108715 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.108682    2554 scope.go:117] "RemoveContainer" containerID="786ed6dff2196304e389c3f7ee0a00b34fb3d63e92ff01016f68fb3fedb821aa"
Sep 10 17:15:02.663000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.662970    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:15:02.663407 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.663388    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/1.log"
Sep 10 17:15:02.663518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.663426    2554 generic.go:356] "Generic (PLEG): container finished" podID="b786b74c-bc04-4817-b490-c2d209a18cdb" containerID="9948c1079bc442b04a6a7d73a09da72924230e55ec37f2ec77385720e756bc91" exitCode=255
Sep 10 17:15:02.663518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.663492    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" event={"ID":"b786b74c-bc04-4817-b490-c2d209a18cdb","Type":"ContainerDied","Data":"9948c1079bc442b04a6a7d73a09da72924230e55ec37f2ec77385720e756bc91"}
Sep 10 17:15:02.663653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.663574    2554 scope.go:117] "RemoveContainer" containerID="786ed6dff2196304e389c3f7ee0a00b34fb3d63e92ff01016f68fb3fedb821aa"
Sep 10 17:15:02.663883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.663865    2554 scope.go:117] "RemoveContainer" containerID="9948c1079bc442b04a6a7d73a09da72924230e55ec37f2ec77385720e756bc91"
Sep 10 17:15:02.664056 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:15:02.664039    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-z7qwj_openshift-console-operator(b786b74c-bc04-4817-b490-c2d209a18cdb)\"" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" podUID="b786b74c-bc04-4817-b490-c2d209a18cdb"
Sep 10 17:15:02.905351 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.905317    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:15:02.907672 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.907646    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/098a0f94-e1af-4846-b58b-d4e132ae8d7b-metrics-tls\") pod \"dns-default-8m29g\" (UID: \"098a0f94-e1af-4846-b58b-d4e132ae8d7b\") " pod="openshift-dns/dns-default-8m29g"
Sep 10 17:15:02.939151 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.939095    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2rm4x\""
Sep 10 17:15:02.947076 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:02.947056    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8m29g"
Sep 10 17:15:03.211721 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:03.211445    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8m29g"]
Sep 10 17:15:03.213887 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:03.213860    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098a0f94_e1af_4846_b58b_d4e132ae8d7b.slice/crio-b0de0f72796011cf651b3db32cb7b777e43d0e51b115b8120e77377121b6259e WatchSource:0}: Error finding container b0de0f72796011cf651b3db32cb7b777e43d0e51b115b8120e77377121b6259e: Status 404 returned error can't find the container with id b0de0f72796011cf651b3db32cb7b777e43d0e51b115b8120e77377121b6259e
Sep 10 17:15:03.668852 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:03.668820    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:15:03.670069 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:03.670043    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8m29g" event={"ID":"098a0f94-e1af-4846-b58b-d4e132ae8d7b","Type":"ContainerStarted","Data":"b0de0f72796011cf651b3db32cb7b777e43d0e51b115b8120e77377121b6259e"}
Sep 10 17:15:05.679121 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:05.679086    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8m29g" event={"ID":"098a0f94-e1af-4846-b58b-d4e132ae8d7b","Type":"ContainerStarted","Data":"9e3c0a0af14c7f2cacb24663cbbe79968e5cbf39c0063f7f37135c4f23839052"}
Sep 10 17:15:05.679520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:05.679143    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8m29g" event={"ID":"098a0f94-e1af-4846-b58b-d4e132ae8d7b","Type":"ContainerStarted","Data":"f3b1114074c3762f5c98f281ec71db459f0c0f67af3b6393b64f7244a0d8c677"}
Sep 10 17:15:05.679520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:05.679197    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8m29g"
Sep 10 17:15:05.701613 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:05.701565    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8m29g" podStartSLOduration=65.596209342 podStartE2EDuration="1m7.701544082s" totalImagesPullingTime="2.10533474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:05.700607137 +0000 UTC m=+101.057097557" watchObservedRunningTime="2026-09-10 17:15:05.701544082 +0000 UTC m=+101.058034472"
Sep 10 17:15:06.562250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:06.562220    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q2mpq"
Sep 10 17:15:08.433275 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.433238    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"]
Sep 10 17:15:08.435636 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.435619    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.442953 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.439711    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 17:15:08.442953 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.439821    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 17:15:08.442953 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.440261    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 17:15:08.443855 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.443833    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 17:15:08.451415 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.451392    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"]
Sep 10 17:15:08.487444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.487413    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77bcd77446-rs27j"]
Sep 10 17:15:08.521331 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.521303    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9bbd76d7f-v7d7k"]
Sep 10 17:15:08.523398 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.523384    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.551924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.550145    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c994fa2-08e7-47bb-84db-50b811f8b39d-tmp\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.551924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.550233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c994fa2-08e7-47bb-84db-50b811f8b39d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.551924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.550265    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8b4j\" (UniqueName: \"kubernetes.io/projected/4c994fa2-08e7-47bb-84db-50b811f8b39d-kube-api-access-q8b4j\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.551924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.551773    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9bbd76d7f-v7d7k"]
Sep 10 17:15:08.650880 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.650839    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-installation-pull-secrets\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.650880 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.650882    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c994fa2-08e7-47bb-84db-50b811f8b39d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.651087 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.650903    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8b4j\" (UniqueName: \"kubernetes.io/projected/4c994fa2-08e7-47bb-84db-50b811f8b39d-kube-api-access-q8b4j\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.651087 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.650941    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-trusted-ca\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.651155 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651085    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-bound-sa-token\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.651155 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651121    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c994fa2-08e7-47bb-84db-50b811f8b39d-tmp\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.651155 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651150    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-registry-tls\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.651293 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651174    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgwm\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-kube-api-access-gjgwm\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.651293 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651246    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-ca-trust-extracted\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.651393 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651300    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-image-registry-private-configuration\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.651393 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-registry-certificates\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.651472 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.651429    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c994fa2-08e7-47bb-84db-50b811f8b39d-tmp\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.653468 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.653448    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c994fa2-08e7-47bb-84db-50b811f8b39d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.663510 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.663491    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8b4j\" (UniqueName: \"kubernetes.io/projected/4c994fa2-08e7-47bb-84db-50b811f8b39d-kube-api-access-q8b4j\") pod \"klusterlet-addon-workmgr-6bc8bfcc5d-dblpd\" (UID: \"4c994fa2-08e7-47bb-84db-50b811f8b39d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.746544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.746457    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:08.752150 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752131    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-trusted-ca\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752241 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752170    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-bound-sa-token\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752241 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752191    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-registry-tls\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752241 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752211    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgwm\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-kube-api-access-gjgwm\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752388 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752243    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-ca-trust-extracted\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752614 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752273    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-image-registry-private-configuration\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752755 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752635    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-registry-certificates\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752755 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752682    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-installation-pull-secrets\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.752755 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.752713    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-ca-trust-extracted\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.753361 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.753319    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-trusted-ca\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.753433 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.753372    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-registry-certificates\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.755318 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.755117    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-image-registry-private-configuration\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.755318 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.755278    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-installation-pull-secrets\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.755318 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.755304    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-registry-tls\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.761121 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.761094    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgwm\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-kube-api-access-gjgwm\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.761331 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.761313    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb-bound-sa-token\") pod \"image-registry-9bbd76d7f-v7d7k\" (UID: \"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb\") " pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:08.850763 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:08.850726    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:09.011916 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:09.011844    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"]
Sep 10 17:15:09.015033 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:09.015006    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c994fa2_08e7_47bb_84db_50b811f8b39d.slice/crio-e991c5c7cd5258096b1128ddfb9e7a8af38baf0f9499afe0cdfb58d1be785bde WatchSource:0}: Error finding container e991c5c7cd5258096b1128ddfb9e7a8af38baf0f9499afe0cdfb58d1be785bde: Status 404 returned error can't find the container with id e991c5c7cd5258096b1128ddfb9e7a8af38baf0f9499afe0cdfb58d1be785bde
Sep 10 17:15:09.120976 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:09.120946    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9bbd76d7f-v7d7k"]
Sep 10 17:15:09.123629 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:09.123597    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e1b1c5_8766_4bb5_b4df_7e9a7f5220fb.slice/crio-26b2e5827f55c9422cdcffa1a6a2189a553e6873fc319df03107a0af1779197e WatchSource:0}: Error finding container 26b2e5827f55c9422cdcffa1a6a2189a553e6873fc319df03107a0af1779197e: Status 404 returned error can't find the container with id 26b2e5827f55c9422cdcffa1a6a2189a553e6873fc319df03107a0af1779197e
Sep 10 17:15:09.696373 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:09.696333    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd" event={"ID":"4c994fa2-08e7-47bb-84db-50b811f8b39d","Type":"ContainerStarted","Data":"e991c5c7cd5258096b1128ddfb9e7a8af38baf0f9499afe0cdfb58d1be785bde"}
Sep 10 17:15:09.697747 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:09.697726    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k" event={"ID":"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb","Type":"ContainerStarted","Data":"4f0e16ea108804973bf6411c28ef1cae085839360b01ccbcf55930d25a6242f4"}
Sep 10 17:15:09.697856 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:09.697764    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k" event={"ID":"97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb","Type":"ContainerStarted","Data":"26b2e5827f55c9422cdcffa1a6a2189a553e6873fc319df03107a0af1779197e"}
Sep 10 17:15:09.697901 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:09.697856    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:09.719898 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:09.719851    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k" podStartSLOduration=1.719837707 podStartE2EDuration="1.719837707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:09.719012473 +0000 UTC m=+105.075502873" watchObservedRunningTime="2026-09-10 17:15:09.719837707 +0000 UTC m=+105.076328107"
Sep 10 17:15:11.400387 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:11.400350    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:15:11.400790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:11.400408    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:15:11.400790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:11.400743    2554 scope.go:117] "RemoveContainer" containerID="9948c1079bc442b04a6a7d73a09da72924230e55ec37f2ec77385720e756bc91"
Sep 10 17:15:11.400907 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:15:11.400889    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-z7qwj_openshift-console-operator(b786b74c-bc04-4817-b490-c2d209a18cdb)\"" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" podUID="b786b74c-bc04-4817-b490-c2d209a18cdb"
Sep 10 17:15:13.715816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:13.715780    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd" event={"ID":"4c994fa2-08e7-47bb-84db-50b811f8b39d","Type":"ContainerStarted","Data":"06dae5304e2d31a2c81881f22260f64e75d1815a5c4d7bd47e1db20fc4835df9"}
Sep 10 17:15:13.716254 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:13.715976    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:13.717567 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:13.717545    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd"
Sep 10 17:15:13.745725 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:13.745677    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bc8bfcc5d-dblpd" podStartSLOduration=1.856058709 podStartE2EDuration="5.745663031s" totalImagesPullingTime="3.889604322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:13.744150744 +0000 UTC m=+109.100641143" watchObservedRunningTime="2026-09-10 17:15:13.745663031 +0000 UTC m=+109.102153430"
Sep 10 17:15:15.685488 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:15.685455    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8m29g"
Sep 10 17:15:17.829052 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:17.829001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:15:17.831328 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:17.831307    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4f8197d-f3d1-4bc5-8e68-bb82c19b972b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz8pq\" (UID: \"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b\") " pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:15:18.107911 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:18.107838    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cz8pq"
Sep 10 17:15:18.397560 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:18.397499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cz8pq"]
Sep 10 17:15:18.400338 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:18.400309    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f8197d_f3d1_4bc5_8e68_bb82c19b972b.slice/crio-b5d0f1cc4fd4121996b3e21292057d7a5295506c5cd498b8249679cf73fa5bf2 WatchSource:0}: Error finding container b5d0f1cc4fd4121996b3e21292057d7a5295506c5cd498b8249679cf73fa5bf2: Status 404 returned error can't find the container with id b5d0f1cc4fd4121996b3e21292057d7a5295506c5cd498b8249679cf73fa5bf2
Sep 10 17:15:18.493635 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:18.493608    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:15:18.733744 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:18.733660    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz8pq" event={"ID":"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b","Type":"ContainerStarted","Data":"b2bf67543036feba8572d8a992861b86950afbe5d490deaa5ccea86d45c288d1"}
Sep 10 17:15:18.733744 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:18.733717    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz8pq" event={"ID":"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b","Type":"ContainerStarted","Data":"b5d0f1cc4fd4121996b3e21292057d7a5295506c5cd498b8249679cf73fa5bf2"}
Sep 10 17:15:19.739069 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:19.738983    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz8pq" event={"ID":"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b","Type":"ContainerStarted","Data":"b335244662fe7027a58b7df64d7e05af08c2f12f4822ed933df6edaef735662f"}
Sep 10 17:15:20.748797 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:20.748757    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz8pq" event={"ID":"f4f8197d-f3d1-4bc5-8e68-bb82c19b972b","Type":"ContainerStarted","Data":"2d0fb729649d705b5b50451bb2285e423c963f187d5ebd79827bd5600d434556"}
Sep 10 17:15:22.108659 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:22.108625    2554 scope.go:117] "RemoveContainer" containerID="9948c1079bc442b04a6a7d73a09da72924230e55ec37f2ec77385720e756bc91"
Sep 10 17:15:22.109114 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:15:22.108855    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-z7qwj_openshift-console-operator(b786b74c-bc04-4817-b490-c2d209a18cdb)\"" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" podUID="b786b74c-bc04-4817-b490-c2d209a18cdb"
Sep 10 17:15:24.728332 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.728279    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cz8pq" podStartSLOduration=37.701149014 podStartE2EDuration="39.728260726s" totalImagesPullingTime="2.027111712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:14:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:20.782584075 +0000 UTC m=+116.139074475" watchObservedRunningTime="2026-09-10 17:15:24.728260726 +0000 UTC m=+120.084751126"
Sep 10 17:15:24.728739 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.728663    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9c2g9"]
Sep 10 17:15:24.730922 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.730906    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.735700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.735682    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 17:15:24.736054 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.736039    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 17:15:24.737741 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.737136    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 17:15:24.737741 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.737230    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 17:15:24.737741 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.737246    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 17:15:24.737741 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.737295    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 17:15:24.737741 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.737350    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gkjjw\""
Sep 10 17:15:24.887723 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887691    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-accelerators-collector-config\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.887723 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887727    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-root\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.887919 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887757    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-wtmp\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.887919 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.887919 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-tls\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.887919 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-textfile\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.888064 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887928    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-metrics-client-ca\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.888064 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9sdw\" (UniqueName: \"kubernetes.io/projected/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-kube-api-access-k9sdw\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.888064 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.887993    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-sys\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988736 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988660    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-metrics-client-ca\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988736 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9sdw\" (UniqueName: \"kubernetes.io/projected/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-kube-api-access-k9sdw\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988736 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988712    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-sys\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988765    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-sys\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988851    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-accelerators-collector-config\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988882    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-root\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988906    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-wtmp\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.988927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988921    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.989138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.988975    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-tls\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.989138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.989013    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-root\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.989138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.989054    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-textfile\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.989138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.989115    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-wtmp\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.989295 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.989279    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-textfile\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:24.996275 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.996250    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 17:15:24.996384 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.996321    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 17:15:24.996384 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.996252    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 17:15:24.996384 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.996370    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 17:15:24.999856 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.999834    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-metrics-client-ca\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:25.000022 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:24.999859    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-accelerators-collector-config\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:25.001604 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.001584    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:25.001685 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.001605    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-node-exporter-tls\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:25.006301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.006281    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 17:15:25.019047 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.019031    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 17:15:25.026764 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.026742    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9sdw\" (UniqueName: \"kubernetes.io/projected/f5bc03b2-3f00-4aa7-b90c-82a09fb0e110-kube-api-access-k9sdw\") pod \"node-exporter-9c2g9\" (UID: \"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110\") " pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:25.051826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.051805    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gkjjw\""
Sep 10 17:15:25.059987 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.059962    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9c2g9"
Sep 10 17:15:25.070071 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:25.070051    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bc03b2_3f00_4aa7_b90c_82a09fb0e110.slice/crio-a06de31da6aa9992724b2b5dae5a41a71d7bf72dc992d4ea0b19ff8c24ba5327 WatchSource:0}: Error finding container a06de31da6aa9992724b2b5dae5a41a71d7bf72dc992d4ea0b19ff8c24ba5327: Status 404 returned error can't find the container with id a06de31da6aa9992724b2b5dae5a41a71d7bf72dc992d4ea0b19ff8c24ba5327
Sep 10 17:15:25.766549 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:25.766490    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c2g9" event={"ID":"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110","Type":"ContainerStarted","Data":"a06de31da6aa9992724b2b5dae5a41a71d7bf72dc992d4ea0b19ff8c24ba5327"}
Sep 10 17:15:26.771335 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:26.771298    2554 generic.go:356] "Generic (PLEG): container finished" podID="f5bc03b2-3f00-4aa7-b90c-82a09fb0e110" containerID="7e196ecc93c21a8c86f0cb4d8a494f83b1e025c952a7926e0d8525824ad9cf6e" exitCode=0
Sep 10 17:15:26.771821 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:26.771347    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c2g9" event={"ID":"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110","Type":"ContainerDied","Data":"7e196ecc93c21a8c86f0cb4d8a494f83b1e025c952a7926e0d8525824ad9cf6e"}
Sep 10 17:15:27.777255 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:27.777216    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c2g9" event={"ID":"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110","Type":"ContainerStarted","Data":"00aaa0aa86fd16b44e92e773e1c58c941f6567785009d85dd26c52738817e2ae"}
Sep 10 17:15:27.777652 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:27.777279    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9c2g9" event={"ID":"f5bc03b2-3f00-4aa7-b90c-82a09fb0e110","Type":"ContainerStarted","Data":"f8642586002e8bbd513382d579cc98dcc2ce02c8448ab501192be73d6b6f16a0"}
Sep 10 17:15:27.828999 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:27.828942    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9c2g9" podStartSLOduration=2.935702827 podStartE2EDuration="3.828926751s" totalImagesPullingTime="829.775681ms" totalInitContainerRuntime="63.448243ms" isStatefulPod=true podCreationTimestamp="2026-09-10 17:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:27.828547277 +0000 UTC m=+123.185037671" watchObservedRunningTime="2026-09-10 17:15:27.828926751 +0000 UTC m=+123.185417151"
Sep 10 17:15:29.310811 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.310778    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6988d857c9-dfgng"]
Sep 10 17:15:29.313434 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.313416    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.319900 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.319873    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 17:15:29.320038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.319885    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 17:15:29.320038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.319914    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 17:15:29.320038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.320018    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-23v8rvndnbip8\""
Sep 10 17:15:29.320243 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.320089    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 17:15:29.320243 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.320197    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xp7qt\""
Sep 10 17:15:29.325410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.325390    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6827ab9-4080-4d45-841b-11630b5e25ba-metrics-server-audit-profiles\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.325496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.325435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsw2\" (UniqueName: \"kubernetes.io/projected/f6827ab9-4080-4d45-841b-11630b5e25ba-kube-api-access-hwsw2\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.325496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.325453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-client-ca-bundle\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.325496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.325484    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-secret-metrics-server-tls\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.325618 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.325497    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6827ab9-4080-4d45-841b-11630b5e25ba-audit-log\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.325618 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.325543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-secret-metrics-server-client-certs\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.325618 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.325582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6827ab9-4080-4d45-841b-11630b5e25ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.349049 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.349026    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6988d857c9-dfgng"]
Sep 10 17:15:29.425873 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.425843    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-secret-metrics-server-tls\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.425991 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.425876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6827ab9-4080-4d45-841b-11630b5e25ba-audit-log\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.425991 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.425908    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-secret-metrics-server-client-certs\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.425991 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.425944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6827ab9-4080-4d45-841b-11630b5e25ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.425991 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.425979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6827ab9-4080-4d45-841b-11630b5e25ba-metrics-server-audit-profiles\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.426131 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.426037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsw2\" (UniqueName: \"kubernetes.io/projected/f6827ab9-4080-4d45-841b-11630b5e25ba-kube-api-access-hwsw2\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.426131 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.426062    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-client-ca-bundle\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.426338 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.426316    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6827ab9-4080-4d45-841b-11630b5e25ba-audit-log\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.426717 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.426691    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6827ab9-4080-4d45-841b-11630b5e25ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.426930 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.426911    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6827ab9-4080-4d45-841b-11630b5e25ba-metrics-server-audit-profiles\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.428428 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.428410    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-secret-metrics-server-tls\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.428833 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.428814    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-client-ca-bundle\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.428945 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.428928    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6827ab9-4080-4d45-841b-11630b5e25ba-secret-metrics-server-client-certs\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.473270 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.473239    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsw2\" (UniqueName: \"kubernetes.io/projected/f6827ab9-4080-4d45-841b-11630b5e25ba-kube-api-access-hwsw2\") pod \"metrics-server-6988d857c9-dfgng\" (UID: \"f6827ab9-4080-4d45-841b-11630b5e25ba\") " pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.622467 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.622378    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:29.896234 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:29.896157    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6988d857c9-dfgng"]
Sep 10 17:15:29.898825 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:29.898798    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6827ab9_4080_4d45_841b_11630b5e25ba.slice/crio-5c5487b7adcf86897eae1d2a729a45046129aab1899d777cdef5aeb342318830 WatchSource:0}: Error finding container 5c5487b7adcf86897eae1d2a729a45046129aab1899d777cdef5aeb342318830: Status 404 returned error can't find the container with id 5c5487b7adcf86897eae1d2a729a45046129aab1899d777cdef5aeb342318830
Sep 10 17:15:30.707077 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:30.707048    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9bbd76d7f-v7d7k"
Sep 10 17:15:30.790037 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:30.789998    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng" event={"ID":"f6827ab9-4080-4d45-841b-11630b5e25ba","Type":"ContainerStarted","Data":"5c5487b7adcf86897eae1d2a729a45046129aab1899d777cdef5aeb342318830"}
Sep 10 17:15:31.756163 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.756129    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:15:31.758808 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.758788    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.765505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.765395    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 17:15:31.765505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.765407    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 17:15:31.765505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.765483    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 17:15:31.765505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.765459    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 17:15:31.766809 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.766784    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qhm29\""
Sep 10 17:15:31.766809 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.766807    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 17:15:31.766970 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.766903    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 17:15:31.766970 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.766907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 17:15:31.767222 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.767205    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 17:15:31.767288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.767250    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 17:15:31.767373 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.767357    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 17:15:31.767458 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.767363    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 17:15:31.767621 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.767605    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cq5p9rpsjb984\""
Sep 10 17:15:31.767718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.767639    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 17:15:31.768723 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.768705    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 17:15:31.774614 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.774598    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 17:15:31.786092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.786070    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:15:31.797185 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.797160    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng" event={"ID":"f6827ab9-4080-4d45-841b-11630b5e25ba","Type":"ContainerStarted","Data":"26232115e43c9f654b63fae95d73144b35003617308824a86838de5db11f4a5a"}
Sep 10 17:15:31.842505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842469    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842503    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842704 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842704 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842704 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842704 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842670    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842704 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842700    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842929 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842732    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.842929 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842870    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7hh\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-kube-api-access-7g7hh\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843027 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842928    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843027 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.842972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843027 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843020    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843046    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843071    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843095    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843122    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-config\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843153    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843370 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843181    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843370 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843210    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.843370 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.843244    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.846376 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.846337    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng" podStartSLOduration=1.645542427 podStartE2EDuration="2.846324565s" totalImagesPullingTime="1.200782138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:31.844854305 +0000 UTC m=+127.201344704" watchObservedRunningTime="2026-09-10 17:15:31.846324565 +0000 UTC m=+127.202814965"
Sep 10 17:15:31.944254 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944254 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944257    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944289    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944305    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944319    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944335    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-config\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944380    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944410    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944439    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944548    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944572    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944601    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944624    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944663    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944739    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944767    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.944891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944823    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.945385 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.945006    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.945385 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.944810    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7hh\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-kube-api-access-7g7hh\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.945385 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.945225    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.945385 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.945269    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.946217 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.946190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.947208 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.947099    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.947645 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.947420    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.948101 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.948077    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.948312 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.948289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.948843 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.948815    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.948937 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.948874    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.949180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.949156    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.949301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.949275    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.949786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.949753    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.950217 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.950193    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.950488 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.950465    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-config\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.950777 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.950749    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.950777 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.950762    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.951114 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.951093    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:31.953024 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:31.953003    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7hh\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-kube-api-access-7g7hh\") pod \"prometheus-k8s-0\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:32.067952 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:32.067920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:32.366224 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:32.366134    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:15:32.372178 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:32.372150    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffe588c_f487_46db_8ce9_cddef18f4806.slice/crio-7b0fa32ec9bc36bc966b03f6a2b2d5c1eed6af0a33f98a26c460dfe4546bb180 WatchSource:0}: Error finding container 7b0fa32ec9bc36bc966b03f6a2b2d5c1eed6af0a33f98a26c460dfe4546bb180: Status 404 returned error can't find the container with id 7b0fa32ec9bc36bc966b03f6a2b2d5c1eed6af0a33f98a26c460dfe4546bb180
Sep 10 17:15:32.803421 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:32.803369    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerStarted","Data":"7b0fa32ec9bc36bc966b03f6a2b2d5c1eed6af0a33f98a26c460dfe4546bb180"}
Sep 10 17:15:33.506436 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.506302    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77bcd77446-rs27j" podUID="e1c5d94e-d9ff-454c-a813-3a5a78e1c227" containerName="registry" containerID="cri-o://5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21" gracePeriod=30
Sep 10 17:15:33.740239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.740215    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:15:33.760000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.759938    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-trusted-ca\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.759971    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760267 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760012    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-certificates\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760267 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760041    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-bound-sa-token\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760267 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760068    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-image-registry-private-configuration\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760267 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760121    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-installation-pull-secrets\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760267 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760169    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnpz6\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-kube-api-access-rnpz6\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760267 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760204    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-ca-trust-extracted\") pod \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\" (UID: \"e1c5d94e-d9ff-454c-a813-3a5a78e1c227\") "
Sep 10 17:15:33.760570 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760360    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:15:33.760570 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760519    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:15:33.760684 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.760499    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-trusted-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.762693 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.762647    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:15:33.762799 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.762714    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:15:33.762799 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.762724    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:15:33.762888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.762796    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-kube-api-access-rnpz6" (OuterVolumeSpecName: "kube-api-access-rnpz6") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "kube-api-access-rnpz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:15:33.762888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.762873    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:15:33.768759 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.768737    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e1c5d94e-d9ff-454c-a813-3a5a78e1c227" (UID: "e1c5d94e-d9ff-454c-a813-3a5a78e1c227"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:15:33.808456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.808419    2554 generic.go:356] "Generic (PLEG): container finished" podID="e1c5d94e-d9ff-454c-a813-3a5a78e1c227" containerID="5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21" exitCode=0
Sep 10 17:15:33.808895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.808471    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bcd77446-rs27j"
Sep 10 17:15:33.808895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.808500    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bcd77446-rs27j" event={"ID":"e1c5d94e-d9ff-454c-a813-3a5a78e1c227","Type":"ContainerDied","Data":"5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21"}
Sep 10 17:15:33.808895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.808560    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bcd77446-rs27j" event={"ID":"e1c5d94e-d9ff-454c-a813-3a5a78e1c227","Type":"ContainerDied","Data":"5b3a75e4dfbd925edcc338ae4aa32c7850d26dedbda4c49383131545c331917a"}
Sep 10 17:15:33.808895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.808578    2554 scope.go:117] "RemoveContainer" containerID="5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21"
Sep 10 17:15:33.810292 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.810257    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba" exitCode=0
Sep 10 17:15:33.810407 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.810315    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"}
Sep 10 17:15:33.818310 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.818287    2554 scope.go:117] "RemoveContainer" containerID="5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21"
Sep 10 17:15:33.818594 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:15:33.818571    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21\": container with ID starting with 5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21 not found: ID does not exist" containerID="5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21"
Sep 10 17:15:33.818699 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.818606    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21"} err="failed to get container status \"5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21\": rpc error: code = NotFound desc = could not find container \"5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21\": container with ID starting with 5d6c85c5a5ae788bba73b11ae9c21c8932e1eb4be7c0fe3b72ce5611551daf21 not found: ID does not exist"
Sep 10 17:15:33.862568 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:15:33.862893 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862868    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-installation-pull-secrets\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.863007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862896    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnpz6\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-kube-api-access-rnpz6\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.863007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862909    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-ca-trust-extracted\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.863007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862919    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.863007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862928    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-registry-certificates\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.863007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862940    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-bound-sa-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.863007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.862951    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1c5d94e-d9ff-454c-a813-3a5a78e1c227-image-registry-private-configuration\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:15:33.864799 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.864780    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1cbbfd7-c7ab-4420-8376-3cbd71ed8829-metrics-certs\") pod \"network-metrics-daemon-n7w82\" (UID: \"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829\") " pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:15:33.869871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.869850    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77bcd77446-rs27j"]
Sep 10 17:15:33.872363 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.872342    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77bcd77446-rs27j"]
Sep 10 17:15:33.933615 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.933586    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n4rqr\""
Sep 10 17:15:33.940736 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:33.940716    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7w82"
Sep 10 17:15:34.209724 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:34.209687    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7w82"]
Sep 10 17:15:34.213345 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:15:34.213316    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cbbfd7_c7ab_4420_8376_3cbd71ed8829.slice/crio-5b555280119dd5df5b0f03b82e98cfc2a2c7f2ff2d1478eb50a107855da3a0cb WatchSource:0}: Error finding container 5b555280119dd5df5b0f03b82e98cfc2a2c7f2ff2d1478eb50a107855da3a0cb: Status 404 returned error can't find the container with id 5b555280119dd5df5b0f03b82e98cfc2a2c7f2ff2d1478eb50a107855da3a0cb
Sep 10 17:15:34.817148 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:34.817111    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7w82" event={"ID":"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829","Type":"ContainerStarted","Data":"5b555280119dd5df5b0f03b82e98cfc2a2c7f2ff2d1478eb50a107855da3a0cb"}
Sep 10 17:15:35.115990 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:35.115964    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1c5d94e-d9ff-454c-a813-3a5a78e1c227" path="/var/lib/kubelet/pods/e1c5d94e-d9ff-454c-a813-3a5a78e1c227/volumes"
Sep 10 17:15:35.824458 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:35.824418    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7w82" event={"ID":"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829","Type":"ContainerStarted","Data":"51288614d5eba3dd7d3558259232c525d71eb63b15ecbe3cac4c20ea540291a1"}
Sep 10 17:15:35.824926 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:35.824519    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7w82" event={"ID":"d1cbbfd7-c7ab-4420-8376-3cbd71ed8829","Type":"ContainerStarted","Data":"55e4d6145ab82873c6a23e59bdc6525defe0fd6619bf206d63c59d06defe8f5c"}
Sep 10 17:15:35.846311 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:35.846263    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n7w82" podStartSLOduration=128.965188618 podStartE2EDuration="2m9.846248362s" totalImagesPullingTime="881.059744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:35.845712172 +0000 UTC m=+131.202202573" watchObservedRunningTime="2026-09-10 17:15:35.846248362 +0000 UTC m=+131.202738766"
Sep 10 17:15:36.835691 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:36.835647    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerStarted","Data":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"}
Sep 10 17:15:36.836161 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:36.835736    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerStarted","Data":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"}
Sep 10 17:15:37.108864 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:37.108832    2554 scope.go:117] "RemoveContainer" containerID="9948c1079bc442b04a6a7d73a09da72924230e55ec37f2ec77385720e756bc91"
Sep 10 17:15:37.843012 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:37.842982    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:15:37.843453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:37.843083    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" event={"ID":"b786b74c-bc04-4817-b490-c2d209a18cdb","Type":"ContainerStarted","Data":"edaecff2e6675f3a85491da83dea8f2528b0d9e4cbefba8e040d13d70430ce06"}
Sep 10 17:15:37.843592 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:37.843562    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:15:37.861088 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:37.861035    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-z7qwj" podStartSLOduration=53.847277569 podStartE2EDuration="56.861018889s" totalImagesPullingTime="3.01374132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:37.859633102 +0000 UTC m=+133.216123527" watchObservedRunningTime="2026-09-10 17:15:37.861018889 +0000 UTC m=+133.217509292"
Sep 10 17:15:38.097226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:38.097151    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-z7qwj"
Sep 10 17:15:38.850055 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:38.850011    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerStarted","Data":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"}
Sep 10 17:15:38.850440 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:38.850135    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerStarted","Data":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"}
Sep 10 17:15:38.850440 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:38.850166    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerStarted","Data":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"}
Sep 10 17:15:38.850440 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:38.850197    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerStarted","Data":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"}
Sep 10 17:15:38.886398 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:38.886338    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6160893339999998 podStartE2EDuration="7.886321351s" totalImagesPullingTime="5.251402893s" totalInitContainerRuntime="18.829124ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:15:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:38.884866312 +0000 UTC m=+134.241356750" watchObservedRunningTime="2026-09-10 17:15:38.886321351 +0000 UTC m=+134.242812121"
Sep 10 17:15:42.068450 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:42.068416    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:49.622805 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:49.622762    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:49.623255 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:49.622835    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:15:55.915479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:55.915444    2554 generic.go:356] "Generic (PLEG): container finished" podID="b30b0823-6551-4431-91ef-bccae87c4810" containerID="40b8176192111ed4922cf40849abb04fc52664df7fb23dd14fca0be7c4ab274b" exitCode=0
Sep 10 17:15:55.915846 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:55.915525    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c" event={"ID":"b30b0823-6551-4431-91ef-bccae87c4810","Type":"ContainerDied","Data":"40b8176192111ed4922cf40849abb04fc52664df7fb23dd14fca0be7c4ab274b"}
Sep 10 17:15:55.915943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:55.915929    2554 scope.go:117] "RemoveContainer" containerID="40b8176192111ed4922cf40849abb04fc52664df7fb23dd14fca0be7c4ab274b"
Sep 10 17:15:56.921139 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:56.921103    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4dk2c" event={"ID":"b30b0823-6551-4431-91ef-bccae87c4810","Type":"ContainerStarted","Data":"62612ff6cb58f5f508a413cc360f42383faa794a8b06ff07656a889ecf35c015"}
Sep 10 17:15:57.113370 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:15:57.113343    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7mrft_b85b6068-2505-4ed6-9dfa-7e5f9841a3a9/serve-healthcheck-canary/0.log"
Sep 10 17:16:00.935974 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:00.935937    2554 generic.go:356] "Generic (PLEG): container finished" podID="eb3ae1d4-2da4-4635-93cf-51fa19c46dc8" containerID="38e33a6d5203f5fbd9fd05efca6ca1894756624926bf00f767a29d4a249f63a6" exitCode=0
Sep 10 17:16:00.936480 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:00.935997    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn" event={"ID":"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8","Type":"ContainerDied","Data":"38e33a6d5203f5fbd9fd05efca6ca1894756624926bf00f767a29d4a249f63a6"}
Sep 10 17:16:00.936480 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:00.936386    2554 scope.go:117] "RemoveContainer" containerID="38e33a6d5203f5fbd9fd05efca6ca1894756624926bf00f767a29d4a249f63a6"
Sep 10 17:16:01.940971 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:01.940940    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-rrnqn" event={"ID":"eb3ae1d4-2da4-4635-93cf-51fa19c46dc8","Type":"ContainerStarted","Data":"a18625d15272d7a208bd8b8c63954c5df120449894054d6c1d73fb59813972fd"}
Sep 10 17:16:09.629072 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:09.629038    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:16:09.633456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:09.633427    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6988d857c9-dfgng"
Sep 10 17:16:32.068367 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:32.068334    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:32.087266 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:32.087243    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:33.068999 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:33.068970    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:50.127920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.127888    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:16:50.128301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.128173    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="prometheus" containerID="cri-o://1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73" gracePeriod=600
Sep 10 17:16:50.128301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.128197    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy" containerID="cri-o://f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a" gracePeriod=600
Sep 10 17:16:50.128301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.128254    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1" gracePeriod=600
Sep 10 17:16:50.128484 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.128213    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="thanos-sidecar" containerID="cri-o://ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef" gracePeriod=600
Sep 10 17:16:50.128484 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.128299    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-web" containerID="cri-o://8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012" gracePeriod=600
Sep 10 17:16:50.128484 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.128233    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="config-reloader" containerID="cri-o://c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd" gracePeriod=600
Sep 10 17:16:50.376048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.376023    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:50.509057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.508978    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509014    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-metrics-client-ca\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509050    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-tls\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509263 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509241    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g7hh\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-kube-api-access-7g7hh\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509312 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509298    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-0\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509356 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509319    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-web-config\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509356 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509340    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-kube-rbac-proxy\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509364    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-metrics-client-certs\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509388    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-config\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509438    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509439    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509466    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-db\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509500    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-trusted-ca-bundle\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509554    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-serving-certs-ca-bundle\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509582    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-thanos-prometheus-http-client-file\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509608    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-2\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509629    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-tls-assets\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509668    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-grpc-tls\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.509696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509697    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-config-out\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.510130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509733    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-kubelet-serving-ca-bundle\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.510130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.509765    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-1\") pod \"6ffe588c-f487-46db-8ce9-cddef18f4806\" (UID: \"6ffe588c-f487-46db-8ce9-cddef18f4806\") "
Sep 10 17:16:50.510130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.510043    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-metrics-client-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.510659 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.510377    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:16:50.511203 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.510791    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:16:50.511564 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.511520    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:16:50.512333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.511985    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-config" (OuterVolumeSpecName: "config") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.512333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.512058    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:16:50.512333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.512204    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.512572 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.512418    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:16:50.512739 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.512699    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:16:50.512817 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.512800    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-kube-api-access-7g7hh" (OuterVolumeSpecName: "kube-api-access-7g7hh") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "kube-api-access-7g7hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:16:50.512876 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.512824    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:16:50.513316 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.513278    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.513404 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.513377    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.513474 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.513400    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.514102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.514077    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:16:50.514210 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.514191    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.514363 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.514341    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.515109 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.515087    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.515219 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.515194    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-config-out" (OuterVolumeSpecName: "config-out") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:16:50.522352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.522327    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-web-config" (OuterVolumeSpecName: "web-config") pod "6ffe588c-f487-46db-8ce9-cddef18f4806" (UID: "6ffe588c-f487-46db-8ce9-cddef18f4806"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:16:50.610377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610346    2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-grpc-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610371    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-config-out\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610382    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610392    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610402    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610412    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610420    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g7hh\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-kube-api-access-7g7hh\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610430    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610439    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-web-config\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610447    2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-kube-rbac-proxy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610455    2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-metrics-client-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610464    2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-config\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610472    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610480    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-db\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610489    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610497    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610505    2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ffe588c-f487-46db-8ce9-cddef18f4806-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610515    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ffe588c-f487-46db-8ce9-cddef18f4806-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:50.610552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:50.610522    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ffe588c-f487-46db-8ce9-cddef18f4806-tls-assets\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:16:51.123187 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123158    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1" exitCode=0
Sep 10 17:16:51.123187 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123184    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a" exitCode=0
Sep 10 17:16:51.123187 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123190    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012" exitCode=0
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123196    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef" exitCode=0
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123201    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd" exitCode=0
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123207    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73" exitCode=0
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123232    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"}
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123300    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"}
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123326    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"}
Sep 10 17:16:51.123360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123350    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"}
Sep 10 17:16:51.123638 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123368    2554 scope.go:117] "RemoveContainer" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.123638 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123373    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"}
Sep 10 17:16:51.123638 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123396    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"}
Sep 10 17:16:51.123638 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.123419    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ffe588c-f487-46db-8ce9-cddef18f4806","Type":"ContainerDied","Data":"7b0fa32ec9bc36bc966b03f6a2b2d5c1eed6af0a33f98a26c460dfe4546bb180"}
Sep 10 17:16:51.131087 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.130889    2554 scope.go:117] "RemoveContainer" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.137912 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.137898    2554 scope.go:117] "RemoveContainer" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.149483 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.149459    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:16:51.154336 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.154312    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:16:51.156212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.156195    2554 scope.go:117] "RemoveContainer" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.164753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.164728    2554 scope.go:117] "RemoveContainer" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.171107 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.171093    2554 scope.go:117] "RemoveContainer" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.178217 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.178200    2554 scope.go:117] "RemoveContainer" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.184423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184399    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:16:51.184694 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184679    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-thanos"
Sep 10 17:16:51.184734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184696    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-thanos"
Sep 10 17:16:51.184734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184706    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="thanos-sidecar"
Sep 10 17:16:51.184734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184711    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="thanos-sidecar"
Sep 10 17:16:51.184734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184725    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="config-reloader"
Sep 10 17:16:51.184734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184734    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="config-reloader"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184743    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-web"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184748    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-web"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184755    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="prometheus"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184761    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="prometheus"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184767    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="init-config-reloader"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184773    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="init-config-reloader"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184782    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184787    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184801    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c5d94e-d9ff-454c-a813-3a5a78e1c227" containerName="registry"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184809    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c5d94e-d9ff-454c-a813-3a5a78e1c227" containerName="registry"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184848    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-web"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184856    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c5d94e-d9ff-454c-a813-3a5a78e1c227" containerName="registry"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184858    2554 scope.go:117] "RemoveContainer" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.184923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184861    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="prometheus"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184942    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="thanos-sidecar"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184955    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy-thanos"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184968    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="config-reloader"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.184978    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" containerName="kube-rbac-proxy"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:16:51.185158    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": container with ID starting with b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1 not found: ID does not exist" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.185186    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"} err="failed to get container status \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": rpc error: code = NotFound desc = could not find container \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": container with ID starting with b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1 not found: ID does not exist"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.185203    2554 scope.go:117] "RemoveContainer" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:16:51.185468    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": container with ID starting with f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a not found: ID does not exist" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.185489    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"} err="failed to get container status \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": rpc error: code = NotFound desc = could not find container \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": container with ID starting with f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a not found: ID does not exist"
Sep 10 17:16:51.185663 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.185504    2554 scope.go:117] "RemoveContainer" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.186107 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:16:51.185753    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": container with ID starting with 8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012 not found: ID does not exist" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.186107 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.185774    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"} err="failed to get container status \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": rpc error: code = NotFound desc = could not find container \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": container with ID starting with 8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012 not found: ID does not exist"
Sep 10 17:16:51.186107 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.185791    2554 scope.go:117] "RemoveContainer" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.186107 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:16:51.185982    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": container with ID starting with ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef not found: ID does not exist" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.186107 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.185999    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"} err="failed to get container status \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": rpc error: code = NotFound desc = could not find container \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": container with ID starting with ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef not found: ID does not exist"
Sep 10 17:16:51.186107 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.186013    2554 scope.go:117] "RemoveContainer" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.186381 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:16:51.186237    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": container with ID starting with c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd not found: ID does not exist" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.186381 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.186262    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"} err="failed to get container status \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": rpc error: code = NotFound desc = could not find container \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": container with ID starting with c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd not found: ID does not exist"
Sep 10 17:16:51.186381 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.186276    2554 scope.go:117] "RemoveContainer" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.186516 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:16:51.186495    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": container with ID starting with 1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73 not found: ID does not exist" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.186580 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.186514    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"} err="failed to get container status \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": rpc error: code = NotFound desc = could not find container \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": container with ID starting with 1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73 not found: ID does not exist"
Sep 10 17:16:51.186580 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.186556    2554 scope.go:117] "RemoveContainer" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.186825 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:16:51.186806    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": container with ID starting with 2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba not found: ID does not exist" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.186863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.186829    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"} err="failed to get container status \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": rpc error: code = NotFound desc = could not find container \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": container with ID starting with 2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba not found: ID does not exist"
Sep 10 17:16:51.186863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.186844    2554 scope.go:117] "RemoveContainer" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.187113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187093    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"} err="failed to get container status \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": rpc error: code = NotFound desc = could not find container \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": container with ID starting with b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1 not found: ID does not exist"
Sep 10 17:16:51.187188 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187115    2554 scope.go:117] "RemoveContainer" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.187376 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187358    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"} err="failed to get container status \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": rpc error: code = NotFound desc = could not find container \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": container with ID starting with f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a not found: ID does not exist"
Sep 10 17:16:51.187423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187376    2554 scope.go:117] "RemoveContainer" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.187645 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187625    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"} err="failed to get container status \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": rpc error: code = NotFound desc = could not find container \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": container with ID starting with 8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012 not found: ID does not exist"
Sep 10 17:16:51.187695 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187646    2554 scope.go:117] "RemoveContainer" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.187874 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187856    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"} err="failed to get container status \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": rpc error: code = NotFound desc = could not find container \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": container with ID starting with ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef not found: ID does not exist"
Sep 10 17:16:51.187978 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.187875    2554 scope.go:117] "RemoveContainer" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.188104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188088    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"} err="failed to get container status \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": rpc error: code = NotFound desc = could not find container \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": container with ID starting with c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd not found: ID does not exist"
Sep 10 17:16:51.188146 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188104    2554 scope.go:117] "RemoveContainer" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.188316 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188300    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"} err="failed to get container status \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": rpc error: code = NotFound desc = could not find container \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": container with ID starting with 1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73 not found: ID does not exist"
Sep 10 17:16:51.188358 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188316    2554 scope.go:117] "RemoveContainer" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.188511 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188491    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"} err="failed to get container status \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": rpc error: code = NotFound desc = could not find container \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": container with ID starting with 2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba not found: ID does not exist"
Sep 10 17:16:51.188511 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188509    2554 scope.go:117] "RemoveContainer" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.188791 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188769    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"} err="failed to get container status \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": rpc error: code = NotFound desc = could not find container \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": container with ID starting with b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1 not found: ID does not exist"
Sep 10 17:16:51.188837 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.188793    2554 scope.go:117] "RemoveContainer" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.189042 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.189020    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"} err="failed to get container status \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": rpc error: code = NotFound desc = could not find container \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": container with ID starting with f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a not found: ID does not exist"
Sep 10 17:16:51.189042 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.189041    2554 scope.go:117] "RemoveContainer" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.189147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.189135    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.189245 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.189227    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"} err="failed to get container status \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": rpc error: code = NotFound desc = could not find container \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": container with ID starting with 8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012 not found: ID does not exist"
Sep 10 17:16:51.189310 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.189248    2554 scope.go:117] "RemoveContainer" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.189789 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.189769    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"} err="failed to get container status \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": rpc error: code = NotFound desc = could not find container \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": container with ID starting with ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef not found: ID does not exist"
Sep 10 17:16:51.189789 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.189789    2554 scope.go:117] "RemoveContainer" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.190059 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190039    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"} err="failed to get container status \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": rpc error: code = NotFound desc = could not find container \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": container with ID starting with c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd not found: ID does not exist"
Sep 10 17:16:51.190118 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190060    2554 scope.go:117] "RemoveContainer" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.190297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190280    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"} err="failed to get container status \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": rpc error: code = NotFound desc = could not find container \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": container with ID starting with 1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73 not found: ID does not exist"
Sep 10 17:16:51.190339 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190298    2554 scope.go:117] "RemoveContainer" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.190506 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190490    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"} err="failed to get container status \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": rpc error: code = NotFound desc = could not find container \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": container with ID starting with 2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba not found: ID does not exist"
Sep 10 17:16:51.190565 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190506    2554 scope.go:117] "RemoveContainer" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.190743 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190728    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"} err="failed to get container status \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": rpc error: code = NotFound desc = could not find container \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": container with ID starting with b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1 not found: ID does not exist"
Sep 10 17:16:51.190743 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190743    2554 scope.go:117] "RemoveContainer" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.190944 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190928    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"} err="failed to get container status \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": rpc error: code = NotFound desc = could not find container \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": container with ID starting with f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a not found: ID does not exist"
Sep 10 17:16:51.190985 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.190944    2554 scope.go:117] "RemoveContainer" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.191160 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191141    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"} err="failed to get container status \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": rpc error: code = NotFound desc = could not find container \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": container with ID starting with 8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012 not found: ID does not exist"
Sep 10 17:16:51.191209 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191161    2554 scope.go:117] "RemoveContainer" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.191403 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191379    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"} err="failed to get container status \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": rpc error: code = NotFound desc = could not find container \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": container with ID starting with ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef not found: ID does not exist"
Sep 10 17:16:51.191461 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191404    2554 scope.go:117] "RemoveContainer" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.191677 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191656    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"} err="failed to get container status \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": rpc error: code = NotFound desc = could not find container \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": container with ID starting with c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd not found: ID does not exist"
Sep 10 17:16:51.191677 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191677    2554 scope.go:117] "RemoveContainer" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.191943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191924    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"} err="failed to get container status \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": rpc error: code = NotFound desc = could not find container \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": container with ID starting with 1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73 not found: ID does not exist"
Sep 10 17:16:51.191943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.191943    2554 scope.go:117] "RemoveContainer" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.192226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192199    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"} err="failed to get container status \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": rpc error: code = NotFound desc = could not find container \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": container with ID starting with 2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba not found: ID does not exist"
Sep 10 17:16:51.192325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192234    2554 scope.go:117] "RemoveContainer" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.192325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192214    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 17:16:51.192325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192214    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cq5p9rpsjb984\""
Sep 10 17:16:51.192473 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192410    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 17:16:51.192473 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192412    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 17:16:51.192594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192481    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 17:16:51.192594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192564    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 17:16:51.192681 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192629    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 17:16:51.192681 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192634    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 17:16:51.192826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192790    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"} err="failed to get container status \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": rpc error: code = NotFound desc = could not find container \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": container with ID starting with b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1 not found: ID does not exist"
Sep 10 17:16:51.192826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192819    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 17:16:51.192967 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192821    2554 scope.go:117] "RemoveContainer" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.192967 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192845    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 17:16:51.192967 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192807    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 17:16:51.192967 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.192964    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 17:16:51.193230 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.193211    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qhm29\""
Sep 10 17:16:51.193316 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.193296    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"} err="failed to get container status \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": rpc error: code = NotFound desc = could not find container \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": container with ID starting with f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a not found: ID does not exist"
Sep 10 17:16:51.193373 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.193320    2554 scope.go:117] "RemoveContainer" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.193608 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.193583    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 17:16:51.193777 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.193741    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"} err="failed to get container status \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": rpc error: code = NotFound desc = could not find container \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": container with ID starting with 8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012 not found: ID does not exist"
Sep 10 17:16:51.193878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.193778    2554 scope.go:117] "RemoveContainer" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.194111 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194087    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"} err="failed to get container status \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": rpc error: code = NotFound desc = could not find container \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": container with ID starting with ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef not found: ID does not exist"
Sep 10 17:16:51.194178 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194113    2554 scope.go:117] "RemoveContainer" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.194380 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194358    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"} err="failed to get container status \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": rpc error: code = NotFound desc = could not find container \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": container with ID starting with c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd not found: ID does not exist"
Sep 10 17:16:51.194380 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194380    2554 scope.go:117] "RemoveContainer" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.194681 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194650    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"} err="failed to get container status \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": rpc error: code = NotFound desc = could not find container \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": container with ID starting with 1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73 not found: ID does not exist"
Sep 10 17:16:51.194681 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194673    2554 scope.go:117] "RemoveContainer" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.194904 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194885    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"} err="failed to get container status \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": rpc error: code = NotFound desc = could not find container \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": container with ID starting with 2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba not found: ID does not exist"
Sep 10 17:16:51.194904 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.194905    2554 scope.go:117] "RemoveContainer" containerID="b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"
Sep 10 17:16:51.195147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.195129    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1"} err="failed to get container status \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": rpc error: code = NotFound desc = could not find container \"b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1\": container with ID starting with b0ee77b1564ed30bee511b79b267b8f9279782b2fb2aab8b70bf3cb13847cef1 not found: ID does not exist"
Sep 10 17:16:51.195229 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.195148    2554 scope.go:117] "RemoveContainer" containerID="f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"
Sep 10 17:16:51.195516 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.195489    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a"} err="failed to get container status \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": rpc error: code = NotFound desc = could not find container \"f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a\": container with ID starting with f5b1a316115cce06aaf59dfde672abc07f4435d0e10e3943f3fa7db22c98907a not found: ID does not exist"
Sep 10 17:16:51.195516 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.195515    2554 scope.go:117] "RemoveContainer" containerID="8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"
Sep 10 17:16:51.195834 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.195800    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012"} err="failed to get container status \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": rpc error: code = NotFound desc = could not find container \"8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012\": container with ID starting with 8b6caa05bdfe43954ccb1a157657ca02d9e657b4f683528cea920fd7eb6da012 not found: ID does not exist"
Sep 10 17:16:51.195834 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.195829    2554 scope.go:117] "RemoveContainer" containerID="ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"
Sep 10 17:16:51.196159 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.196132    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef"} err="failed to get container status \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": rpc error: code = NotFound desc = could not find container \"ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef\": container with ID starting with ef078e895ea02cbad14a2755c02d987f76d006612ee2cd7059f384240522f6ef not found: ID does not exist"
Sep 10 17:16:51.196235 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.196161    2554 scope.go:117] "RemoveContainer" containerID="c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"
Sep 10 17:16:51.196633 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.196480    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd"} err="failed to get container status \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": rpc error: code = NotFound desc = could not find container \"c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd\": container with ID starting with c41e8ea5a9775af53116f9b0814eacd9d4b2ba570e9c6f8879e14bd9a3f0a2bd not found: ID does not exist"
Sep 10 17:16:51.196633 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.196511    2554 scope.go:117] "RemoveContainer" containerID="1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"
Sep 10 17:16:51.196754 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.196714    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 17:16:51.196895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.196874    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73"} err="failed to get container status \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": rpc error: code = NotFound desc = could not find container \"1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73\": container with ID starting with 1b6ac7cdcf4bbe4e8505a2271932232b6fbc1697c79eda5c0c5ce7836280dd73 not found: ID does not exist"
Sep 10 17:16:51.196958 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.196896    2554 scope.go:117] "RemoveContainer" containerID="2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"
Sep 10 17:16:51.197152 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.197134    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba"} err="failed to get container status \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": rpc error: code = NotFound desc = could not find container \"2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba\": container with ID starting with 2c9522cb08b115325d47a66e657908653f536af22fe2486f2650657ad30bb3ba not found: ID does not exist"
Sep 10 17:16:51.200158 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.200140    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 17:16:51.206175 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.206157    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:16:51.315597 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315552    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-config\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.315597 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315601    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.315820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315648    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.315820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315667    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.315820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315755    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.315820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315791    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316003 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315823    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316003 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316003 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315859    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-web-config\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316003 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315891    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316003 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.315975    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316004    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/100f559f-2a2a-4980-adcc-4c54bad7b39c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316101    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316128    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316169    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmff\" (UniqueName: \"kubernetes.io/projected/100f559f-2a2a-4980-adcc-4c54bad7b39c-kube-api-access-sfmff\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316241    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316264    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.316408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.316282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/100f559f-2a2a-4980-adcc-4c54bad7b39c-config-out\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.416946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.416857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.416946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.416891    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.416946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.416914    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417079    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417099    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-web-config\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417118    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417143    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417162    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/100f559f-2a2a-4980-adcc-4c54bad7b39c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417192    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417226    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417259    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417305    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417337    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmff\" (UniqueName: \"kubernetes.io/projected/100f559f-2a2a-4980-adcc-4c54bad7b39c-kube-api-access-sfmff\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417365    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417391    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417421    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/100f559f-2a2a-4980-adcc-4c54bad7b39c-config-out\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417462    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-config\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417483    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417944 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417487    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417944 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417579    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417944 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417619    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417944 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417624    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.417944 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.417685    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.418177 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.418139    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.418177 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.418143    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.419758 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.418804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.419758 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.419662    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.420220 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.420198    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.420306 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.420219    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.420306 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.420246    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.420656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.420629    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/100f559f-2a2a-4980-adcc-4c54bad7b39c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.421366 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.421289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-web-config\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.421479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.421455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.421891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.421870    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.422307 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.422288    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/100f559f-2a2a-4980-adcc-4c54bad7b39c-config-out\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.422880 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.422854    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.422965 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.422889    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.423205 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.423185    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/100f559f-2a2a-4980-adcc-4c54bad7b39c-config\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.423348 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.423331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/100f559f-2a2a-4980-adcc-4c54bad7b39c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.426124 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.426100    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmff\" (UniqueName: \"kubernetes.io/projected/100f559f-2a2a-4980-adcc-4c54bad7b39c-kube-api-access-sfmff\") pod \"prometheus-k8s-0\" (UID: \"100f559f-2a2a-4980-adcc-4c54bad7b39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.499325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.499297    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:51.782787 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:51.782711    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:16:51.785561 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:16:51.785521    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f559f_2a2a_4980_adcc_4c54bad7b39c.slice/crio-72fd4004928747844373386711d8881577e4b80e0fa3b212adb90993a09ed506 WatchSource:0}: Error finding container 72fd4004928747844373386711d8881577e4b80e0fa3b212adb90993a09ed506: Status 404 returned error can't find the container with id 72fd4004928747844373386711d8881577e4b80e0fa3b212adb90993a09ed506
Sep 10 17:16:52.132695 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:52.132656    2554 generic.go:356] "Generic (PLEG): container finished" podID="100f559f-2a2a-4980-adcc-4c54bad7b39c" containerID="bbe7a837a54c22b8c686b73fcabac86df6858a1de89b25945914d40e1d302b72" exitCode=0
Sep 10 17:16:52.133092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:52.132743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerDied","Data":"bbe7a837a54c22b8c686b73fcabac86df6858a1de89b25945914d40e1d302b72"}
Sep 10 17:16:52.133092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:52.132843    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerStarted","Data":"72fd4004928747844373386711d8881577e4b80e0fa3b212adb90993a09ed506"}
Sep 10 17:16:53.113678 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.113642    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ffe588c-f487-46db-8ce9-cddef18f4806" path="/var/lib/kubelet/pods/6ffe588c-f487-46db-8ce9-cddef18f4806/volumes"
Sep 10 17:16:53.141220 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.141190    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerStarted","Data":"4563a2648f41861e06a8b89cc19af7198debe4f95a2575820ab3caca5b10a87f"}
Sep 10 17:16:53.141603 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.141248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerStarted","Data":"e6e726c052fc8317c2d0e094210141267976b8c8a226d369809ffa8a77b7df74"}
Sep 10 17:16:53.141603 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.141276    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerStarted","Data":"14433b03c0e6f458051a2b1b125beabbda39780bab5c21984ea888abeffe722e"}
Sep 10 17:16:53.141603 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.141302    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerStarted","Data":"c1acb11e78695c42ed939cff4f8784e31fcd90b573a46bae53d2b664ec5029c9"}
Sep 10 17:16:53.141603 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.141325    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerStarted","Data":"f82db767d555919f70652d02efeff3ad306ace9d286c51afe07ae34dd2ec1819"}
Sep 10 17:16:53.141603 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.141348    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"100f559f-2a2a-4980-adcc-4c54bad7b39c","Type":"ContainerStarted","Data":"d712c23888c2bea13d5da2b0c74f9a51ba9f487743f17414eea197cc767656a5"}
Sep 10 17:16:53.173249 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:53.173200    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.154104089 podStartE2EDuration="2.173183968s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.079879ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:16:53.170888651 +0000 UTC m=+208.527379053" watchObservedRunningTime="2026-09-10 17:16:53.173183968 +0000 UTC m=+208.529674374"
Sep 10 17:16:56.499681 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:16:56.499642    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:17:51.500255 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:17:51.500168    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:17:51.515794 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:17:51.515769    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:17:52.373680 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:17:52.373651    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:18:24.987414 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:18:24.987371    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:18:24.989226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:18:24.989203    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:18:25.003594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:18:25.003562    2554 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 17:20:13.055100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.055065    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"]
Sep 10 17:20:13.058356 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.058333    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.063007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.062984    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 17:20:13.063847 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.063818    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xgkxt\""
Sep 10 17:20:13.064736 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.064717    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 17:20:13.066541 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.066505    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 17:20:13.066666 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.066649    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 17:20:13.098698 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.098667    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"]
Sep 10 17:20:13.152019 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.151983    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.152195 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.152028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-tmp\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.152195 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.152047    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-manifests\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.152195 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.152152    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.152195 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.152182    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmlq\" (UniqueName: \"kubernetes.io/projected/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-kube-api-access-kjmlq\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.252904 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.252860    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-tmp\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.252904 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.252902    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-manifests\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.253115 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.252947    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.253115 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.252964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmlq\" (UniqueName: \"kubernetes.io/projected/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-kube-api-access-kjmlq\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.253115 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.253032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.253373 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.253349    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-manifests\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.255309 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.255281    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-tmp\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.255481 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.255462    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.255618 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.255502    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.263615 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.263591    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmlq\" (UniqueName: \"kubernetes.io/projected/632520e6-0e12-4727-ba0f-2ce2a2d3ca20-kube-api-access-kjmlq\") pod \"opendatahub-operator-controller-manager-8dc495f6d-vn8sj\" (UID: \"632520e6-0e12-4727-ba0f-2ce2a2d3ca20\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.369077 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.368975    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:13.668744 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.668663    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"]
Sep 10 17:20:13.672900 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:20:13.672875    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632520e6_0e12_4727_ba0f_2ce2a2d3ca20.slice/crio-9c636a468269abf238fbea75a2ff63b14d400037902b19260eed6cd1cb9062df WatchSource:0}: Error finding container 9c636a468269abf238fbea75a2ff63b14d400037902b19260eed6cd1cb9062df: Status 404 returned error can't find the container with id 9c636a468269abf238fbea75a2ff63b14d400037902b19260eed6cd1cb9062df
Sep 10 17:20:13.674627 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.674609    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:20:13.864415 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:13.864378    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj" event={"ID":"632520e6-0e12-4727-ba0f-2ce2a2d3ca20","Type":"ContainerStarted","Data":"9c636a468269abf238fbea75a2ff63b14d400037902b19260eed6cd1cb9062df"}
Sep 10 17:20:16.879768 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:16.879683    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj" event={"ID":"632520e6-0e12-4727-ba0f-2ce2a2d3ca20","Type":"ContainerStarted","Data":"4995c5b52669fffe2d3bf28944388025994843b2f346de8473e061339a303cde"}
Sep 10 17:20:17.886008 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:17.885969    2554 generic.go:356] "Generic (PLEG): container finished" podID="632520e6-0e12-4727-ba0f-2ce2a2d3ca20" containerID="4995c5b52669fffe2d3bf28944388025994843b2f346de8473e061339a303cde" exitCode=0
Sep 10 17:20:17.886377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:17.886035    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj" event={"ID":"632520e6-0e12-4727-ba0f-2ce2a2d3ca20","Type":"ContainerDied","Data":"4995c5b52669fffe2d3bf28944388025994843b2f346de8473e061339a303cde"}
Sep 10 17:20:18.891994 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:18.891958    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj" event={"ID":"632520e6-0e12-4727-ba0f-2ce2a2d3ca20","Type":"ContainerStarted","Data":"ddba37dd968d7f5367409e496406071c9b0bd9043deda079324b16a88ce4c7b0"}
Sep 10 17:20:18.892379 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:18.892084    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:18.935839 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:18.935781    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj" podStartSLOduration=1.618699932 podStartE2EDuration="5.935763856s" totalImagesPullingTime="3.314901807s" totalInitContainerRuntime="1.002162117s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:18.934099654 +0000 UTC m=+414.290590058" watchObservedRunningTime="2026-09-10 17:20:18.935763856 +0000 UTC m=+414.292254255"
Sep 10 17:20:29.897978 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:29.897944    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-vn8sj"
Sep 10 17:20:31.602484 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.602453    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"]
Sep 10 17:20:31.616469 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.616442    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.621520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.621496    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 17:20:31.622347 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.622326    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 17:20:31.622852 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.622691    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ll68g\""
Sep 10 17:20:31.623011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.622992    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:20:31.623106 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.623017    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 17:20:31.623271 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.623253    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 17:20:31.664753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.664725    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"]
Sep 10 17:20:31.809734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.809699    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eaea14d-952f-454e-b5fd-d411d549cb1c-cert\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.809903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.809742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6eaea14d-952f-454e-b5fd-d411d549cb1c-manager-config\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.809903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.809781    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6eaea14d-952f-454e-b5fd-d411d549cb1c-metrics-cert\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.809903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.809812    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqf9\" (UniqueName: \"kubernetes.io/projected/6eaea14d-952f-454e-b5fd-d411d549cb1c-kube-api-access-fqqf9\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.910564 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.910467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6eaea14d-952f-454e-b5fd-d411d549cb1c-metrics-cert\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.910564 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.910505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqf9\" (UniqueName: \"kubernetes.io/projected/6eaea14d-952f-454e-b5fd-d411d549cb1c-kube-api-access-fqqf9\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.910775 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.910572    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eaea14d-952f-454e-b5fd-d411d549cb1c-cert\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.910775 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.910589    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6eaea14d-952f-454e-b5fd-d411d549cb1c-manager-config\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.911198 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.911175    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6eaea14d-952f-454e-b5fd-d411d549cb1c-manager-config\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.912895 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.912872    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6eaea14d-952f-454e-b5fd-d411d549cb1c-metrics-cert\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.913095 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.913075    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6eaea14d-952f-454e-b5fd-d411d549cb1c-cert\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:31.956546 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:31.956513    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqf9\" (UniqueName: \"kubernetes.io/projected/6eaea14d-952f-454e-b5fd-d411d549cb1c-kube-api-access-fqqf9\") pod \"lws-controller-manager-8655464fff-62rkw\" (UID: \"6eaea14d-952f-454e-b5fd-d411d549cb1c\") " pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:32.225360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:32.225276    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:32.554548 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:32.554499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"]
Sep 10 17:20:32.559468 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:20:32.559440    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaea14d_952f_454e_b5fd_d411d549cb1c.slice/crio-ce8348e781847579b349578b5ac42630fe5030392cd290842e460976370804c1 WatchSource:0}: Error finding container ce8348e781847579b349578b5ac42630fe5030392cd290842e460976370804c1: Status 404 returned error can't find the container with id ce8348e781847579b349578b5ac42630fe5030392cd290842e460976370804c1
Sep 10 17:20:32.942544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:32.942450    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw" event={"ID":"6eaea14d-952f-454e-b5fd-d411d549cb1c","Type":"ContainerStarted","Data":"ce8348e781847579b349578b5ac42630fe5030392cd290842e460976370804c1"}
Sep 10 17:20:33.330426 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.330379    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"]
Sep 10 17:20:33.332659 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.332634    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.367762 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.367702    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 17:20:33.367762 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.367735    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 17:20:33.367938 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.367702    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hgzkv\""
Sep 10 17:20:33.369888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.369857    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 17:20:33.370658 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.370641    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 17:20:33.504083 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.504051    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"]
Sep 10 17:20:33.522444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.522412    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d74e93-a104-4a56-a97f-716edd7b1db5-tmp\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.522625 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.522477    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d74e93-a104-4a56-a97f-716edd7b1db5-tls-certs\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.522625 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.522516    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52x8r\" (UniqueName: \"kubernetes.io/projected/f6d74e93-a104-4a56-a97f-716edd7b1db5-kube-api-access-52x8r\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.623818 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.623690    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d74e93-a104-4a56-a97f-716edd7b1db5-tmp\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.623818 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.623766    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d74e93-a104-4a56-a97f-716edd7b1db5-tls-certs\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.623818 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.623810    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52x8r\" (UniqueName: \"kubernetes.io/projected/f6d74e93-a104-4a56-a97f-716edd7b1db5-kube-api-access-52x8r\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.626552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.626508    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d74e93-a104-4a56-a97f-716edd7b1db5-tmp\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.626920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.626868    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d74e93-a104-4a56-a97f-716edd7b1db5-tls-certs\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.744486 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.744449    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52x8r\" (UniqueName: \"kubernetes.io/projected/f6d74e93-a104-4a56-a97f-716edd7b1db5-kube-api-access-52x8r\") pod \"kube-auth-proxy-555bd9596f-jkz7n\" (UID: \"f6d74e93-a104-4a56-a97f-716edd7b1db5\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:33.943978 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:33.943891    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"
Sep 10 17:20:34.275995 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:34.275914    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n"]
Sep 10 17:20:34.280047 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:20:34.280016    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d74e93_a104_4a56_a97f_716edd7b1db5.slice/crio-d650f9662e97c697e59003e2bc5fa9ba58825d723a4d00fdc02e63b4e7cb9770 WatchSource:0}: Error finding container d650f9662e97c697e59003e2bc5fa9ba58825d723a4d00fdc02e63b4e7cb9770: Status 404 returned error can't find the container with id d650f9662e97c697e59003e2bc5fa9ba58825d723a4d00fdc02e63b4e7cb9770
Sep 10 17:20:34.953193 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:34.953153    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n" event={"ID":"f6d74e93-a104-4a56-a97f-716edd7b1db5","Type":"ContainerStarted","Data":"d650f9662e97c697e59003e2bc5fa9ba58825d723a4d00fdc02e63b4e7cb9770"}
Sep 10 17:20:35.962068 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:35.961977    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw" event={"ID":"6eaea14d-952f-454e-b5fd-d411d549cb1c","Type":"ContainerStarted","Data":"81e7e0aadc8371ac16eb5dfa03825c71f96c55b397da14aa136cb3a5822d9410"}
Sep 10 17:20:35.962843 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:35.962792    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:36.019291 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:36.019230    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw" podStartSLOduration=1.972652136 podStartE2EDuration="5.019208961s" totalImagesPullingTime="3.046556825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:36.0187049 +0000 UTC m=+431.375195303" watchObservedRunningTime="2026-09-10 17:20:36.019208961 +0000 UTC m=+431.375699362"
Sep 10 17:20:37.972449 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:37.972416    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n" event={"ID":"f6d74e93-a104-4a56-a97f-716edd7b1db5","Type":"ContainerStarted","Data":"5ca5854cbd7982e09172de28be96ae02d0e6f801ceeeb3b29524c3ad97a68e5b"}
Sep 10 17:20:47.974964 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:47.974890    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8655464fff-62rkw"
Sep 10 17:20:48.043520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:20:48.043461    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-555bd9596f-jkz7n" podStartSLOduration=11.584900904 podStartE2EDuration="15.04344711s" totalImagesPullingTime="3.458546206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:38.100113718 +0000 UTC m=+433.456604117" watchObservedRunningTime="2026-09-10 17:20:48.04344711 +0000 UTC m=+443.399937509"
Sep 10 17:22:35.420570 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.420480    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"]
Sep 10 17:22:35.423674 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.423657    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.427858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.427835    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 17:22:35.428604 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.428586    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 17:22:35.428696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.428628    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 17:22:35.428696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.428632    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 17:22:35.429582 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.429564    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fs6h2\""
Sep 10 17:22:35.439634 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.439610    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"]
Sep 10 17:22:35.478379 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.478347    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d37b5b84-9568-44ec-9acd-22f1257d36ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.478579 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.478388    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8hc\" (UniqueName: \"kubernetes.io/projected/d37b5b84-9568-44ec-9acd-22f1257d36ef-kube-api-access-mt8hc\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.478579 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.478426    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d37b5b84-9568-44ec-9acd-22f1257d36ef-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.578863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.578828    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d37b5b84-9568-44ec-9acd-22f1257d36ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.579041 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.578875    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8hc\" (UniqueName: \"kubernetes.io/projected/d37b5b84-9568-44ec-9acd-22f1257d36ef-kube-api-access-mt8hc\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.579041 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.578908    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d37b5b84-9568-44ec-9acd-22f1257d36ef-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.579041 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:22:35.578930    2554 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 17:22:35.579041 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:22:35.579004    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37b5b84-9568-44ec-9acd-22f1257d36ef-plugin-serving-cert podName:d37b5b84-9568-44ec-9acd-22f1257d36ef nodeName:}" failed. No retries permitted until 2026-09-10 17:22:36.078987428 +0000 UTC m=+551.435477805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d37b5b84-9568-44ec-9acd-22f1257d36ef-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-x8jmm" (UID: "d37b5b84-9568-44ec-9acd-22f1257d36ef") : secret "plugin-serving-cert" not found
Sep 10 17:22:35.579546 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.579512    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d37b5b84-9568-44ec-9acd-22f1257d36ef-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:35.610614 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:35.610581    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8hc\" (UniqueName: \"kubernetes.io/projected/d37b5b84-9568-44ec-9acd-22f1257d36ef-kube-api-access-mt8hc\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:36.082426 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:36.082392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d37b5b84-9568-44ec-9acd-22f1257d36ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:36.084718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:36.084697    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d37b5b84-9568-44ec-9acd-22f1257d36ef-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-x8jmm\" (UID: \"d37b5b84-9568-44ec-9acd-22f1257d36ef\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:36.341288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:36.341207    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"
Sep 10 17:22:36.623208 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:36.620411    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm"]
Sep 10 17:22:36.626745 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:22:36.626699    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd37b5b84_9568_44ec_9acd_22f1257d36ef.slice/crio-2560392a0967f97569ac9b55b27558e7d3a205bb13bd730755a30281041ee899 WatchSource:0}: Error finding container 2560392a0967f97569ac9b55b27558e7d3a205bb13bd730755a30281041ee899: Status 404 returned error can't find the container with id 2560392a0967f97569ac9b55b27558e7d3a205bb13bd730755a30281041ee899
Sep 10 17:22:37.447636 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:37.447571    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm" event={"ID":"d37b5b84-9568-44ec-9acd-22f1257d36ef","Type":"ContainerStarted","Data":"2560392a0967f97569ac9b55b27558e7d3a205bb13bd730755a30281041ee899"}
Sep 10 17:22:41.468065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:41.467980    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm" event={"ID":"d37b5b84-9568-44ec-9acd-22f1257d36ef","Type":"ContainerStarted","Data":"1fb72c02394995a164fde4d9c5849eb13677554ac4022967b29b19786c9ca66a"}
Sep 10 17:22:41.511523 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:22:41.511467    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-x8jmm" podStartSLOduration=2.001662932 podStartE2EDuration="6.511450071s" totalImagesPullingTime="4.509787139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:41.510366764 +0000 UTC m=+556.866857165" watchObservedRunningTime="2026-09-10 17:22:41.511450071 +0000 UTC m=+556.867940471"
Sep 10 17:23:25.019181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.019150    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:23:25.020094 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.020070    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:23:25.248186 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.248142    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:23:25.250250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.250231    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.253837 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.253812    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 17:23:25.275438 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.275376    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:23:25.292853 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.292822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9bdef861-edc3-4c74-bc0a-eb9056e647c7-config-file\") pod \"limitador-limitador-84994f7ffc-pwczx\" (UID: \"9bdef861-edc3-4c74-bc0a-eb9056e647c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.292957 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.292906    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4jq5\" (UniqueName: \"kubernetes.io/projected/9bdef861-edc3-4c74-bc0a-eb9056e647c7-kube-api-access-d4jq5\") pod \"limitador-limitador-84994f7ffc-pwczx\" (UID: \"9bdef861-edc3-4c74-bc0a-eb9056e647c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.293019 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.293005    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:23:25.393646 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.393614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9bdef861-edc3-4c74-bc0a-eb9056e647c7-config-file\") pod \"limitador-limitador-84994f7ffc-pwczx\" (UID: \"9bdef861-edc3-4c74-bc0a-eb9056e647c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.393785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.393667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4jq5\" (UniqueName: \"kubernetes.io/projected/9bdef861-edc3-4c74-bc0a-eb9056e647c7-kube-api-access-d4jq5\") pod \"limitador-limitador-84994f7ffc-pwczx\" (UID: \"9bdef861-edc3-4c74-bc0a-eb9056e647c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.394199 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.394181    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9bdef861-edc3-4c74-bc0a-eb9056e647c7-config-file\") pod \"limitador-limitador-84994f7ffc-pwczx\" (UID: \"9bdef861-edc3-4c74-bc0a-eb9056e647c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.404570 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.404521    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4jq5\" (UniqueName: \"kubernetes.io/projected/9bdef861-edc3-4c74-bc0a-eb9056e647c7-kube-api-access-d4jq5\") pod \"limitador-limitador-84994f7ffc-pwczx\" (UID: \"9bdef861-edc3-4c74-bc0a-eb9056e647c7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.560084 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.560049    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:25.833490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:25.833408    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:23:25.838002 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:23:25.837974    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdef861_edc3_4c74_bc0a_eb9056e647c7.slice/crio-55008d93972d92fdd650de0071cac5f71a2db736f294ba8bc665858ef4a8fdd8 WatchSource:0}: Error finding container 55008d93972d92fdd650de0071cac5f71a2db736f294ba8bc665858ef4a8fdd8: Status 404 returned error can't find the container with id 55008d93972d92fdd650de0071cac5f71a2db736f294ba8bc665858ef4a8fdd8
Sep 10 17:23:26.211776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.211696    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwhsm"]
Sep 10 17:23:26.214313 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.214298    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwhsm"
Sep 10 17:23:26.217899 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.217876    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dsh28\""
Sep 10 17:23:26.226059 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.226039    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwhsm"]
Sep 10 17:23:26.302026 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.301999    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqps7\" (UniqueName: \"kubernetes.io/projected/5ac020bb-ca18-4528-a830-8f2e0452d43e-kube-api-access-wqps7\") pod \"authorino-59c974bd48-qwhsm\" (UID: \"5ac020bb-ca18-4528-a830-8f2e0452d43e\") " pod="kuadrant-system/authorino-59c974bd48-qwhsm"
Sep 10 17:23:26.402999 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.402949    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqps7\" (UniqueName: \"kubernetes.io/projected/5ac020bb-ca18-4528-a830-8f2e0452d43e-kube-api-access-wqps7\") pod \"authorino-59c974bd48-qwhsm\" (UID: \"5ac020bb-ca18-4528-a830-8f2e0452d43e\") " pod="kuadrant-system/authorino-59c974bd48-qwhsm"
Sep 10 17:23:26.415911 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.415887    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqps7\" (UniqueName: \"kubernetes.io/projected/5ac020bb-ca18-4528-a830-8f2e0452d43e-kube-api-access-wqps7\") pod \"authorino-59c974bd48-qwhsm\" (UID: \"5ac020bb-ca18-4528-a830-8f2e0452d43e\") " pod="kuadrant-system/authorino-59c974bd48-qwhsm"
Sep 10 17:23:26.524146 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.524070    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwhsm"
Sep 10 17:23:26.649539 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.649493    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx" event={"ID":"9bdef861-edc3-4c74-bc0a-eb9056e647c7","Type":"ContainerStarted","Data":"55008d93972d92fdd650de0071cac5f71a2db736f294ba8bc665858ef4a8fdd8"}
Sep 10 17:23:26.799323 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:26.799293    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwhsm"]
Sep 10 17:23:26.802340 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:23:26.802306    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ac020bb_ca18_4528_a830_8f2e0452d43e.slice/crio-8e9e329b3259d34a098a140446cd362faa37d29dff116c8b2c3595cb53b4c0b9 WatchSource:0}: Error finding container 8e9e329b3259d34a098a140446cd362faa37d29dff116c8b2c3595cb53b4c0b9: Status 404 returned error can't find the container with id 8e9e329b3259d34a098a140446cd362faa37d29dff116c8b2c3595cb53b4c0b9
Sep 10 17:23:27.657190 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:27.657138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwhsm" event={"ID":"5ac020bb-ca18-4528-a830-8f2e0452d43e","Type":"ContainerStarted","Data":"8e9e329b3259d34a098a140446cd362faa37d29dff116c8b2c3595cb53b4c0b9"}
Sep 10 17:23:29.666750 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:29.666668    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwhsm" event={"ID":"5ac020bb-ca18-4528-a830-8f2e0452d43e","Type":"ContainerStarted","Data":"5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4"}
Sep 10 17:23:29.685747 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:29.685703    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qwhsm" podStartSLOduration=1.171934294 podStartE2EDuration="3.685688198s" totalImagesPullingTime="2.513753904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:29.684099999 +0000 UTC m=+605.040590402" watchObservedRunningTime="2026-09-10 17:23:29.685688198 +0000 UTC m=+605.042178592"
Sep 10 17:23:40.715243 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:40.715210    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx" event={"ID":"9bdef861-edc3-4c74-bc0a-eb9056e647c7","Type":"ContainerStarted","Data":"f033f10be01d8fcd4bf316d5cfe37a347f9560278452dcb76c62cf43bcea89c1"}
Sep 10 17:23:40.715657 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:40.715275    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:51.725660 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:51.725584    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx"
Sep 10 17:23:51.747981 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:23:51.747931    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-pwczx" podStartSLOduration=12.404222565 podStartE2EDuration="26.747916789s" totalImagesPullingTime="14.343694224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:40.766786715 +0000 UTC m=+616.123277114" watchObservedRunningTime="2026-09-10 17:23:51.747916789 +0000 UTC m=+627.104407189"
Sep 10 17:24:09.727744 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.727707    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q5t6b"]
Sep 10 17:24:09.730922 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.730907    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:09.734863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.734844    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 17:24:09.748175 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.748146    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q5t6b"]
Sep 10 17:24:09.759712 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.759678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsg5m\" (UniqueName: \"kubernetes.io/projected/40f6ee63-8acb-4550-ab6d-ec9843024bea-kube-api-access-xsg5m\") pod \"authorino-6d4dfc649f-q5t6b\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") " pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:09.759828 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.759742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40f6ee63-8acb-4550-ab6d-ec9843024bea-tls-cert\") pod \"authorino-6d4dfc649f-q5t6b\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") " pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:09.856609 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.856031    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q5t6b"]
Sep 10 17:24:09.856609 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:24:09.856585    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xsg5m tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-q5t6b" podUID="40f6ee63-8acb-4550-ab6d-ec9843024bea"
Sep 10 17:24:09.860610 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.860578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsg5m\" (UniqueName: \"kubernetes.io/projected/40f6ee63-8acb-4550-ab6d-ec9843024bea-kube-api-access-xsg5m\") pod \"authorino-6d4dfc649f-q5t6b\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") " pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:09.860738 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.860618    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40f6ee63-8acb-4550-ab6d-ec9843024bea-tls-cert\") pod \"authorino-6d4dfc649f-q5t6b\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") " pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:09.863010 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.862994    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40f6ee63-8acb-4550-ab6d-ec9843024bea-tls-cert\") pod \"authorino-6d4dfc649f-q5t6b\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") " pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:09.874871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.874849    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsg5m\" (UniqueName: \"kubernetes.io/projected/40f6ee63-8acb-4550-ab6d-ec9843024bea-kube-api-access-xsg5m\") pod \"authorino-6d4dfc649f-q5t6b\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") " pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:09.909878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.909852    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d58cf989c-47j9c"]
Sep 10 17:24:09.913080 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.913064    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:09.929069 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.929045    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d58cf989c-47j9c"]
Sep 10 17:24:09.960993 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.960963    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39895d50-c298-417f-ac88-7d693255237c-tls-cert\") pod \"authorino-5d58cf989c-47j9c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") " pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:09.961138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:09.961025    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrklz\" (UniqueName: \"kubernetes.io/projected/39895d50-c298-417f-ac88-7d693255237c-kube-api-access-mrklz\") pod \"authorino-5d58cf989c-47j9c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") " pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:10.061600 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.061565    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39895d50-c298-417f-ac88-7d693255237c-tls-cert\") pod \"authorino-5d58cf989c-47j9c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") " pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:10.061776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.061735    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrklz\" (UniqueName: \"kubernetes.io/projected/39895d50-c298-417f-ac88-7d693255237c-kube-api-access-mrklz\") pod \"authorino-5d58cf989c-47j9c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") " pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:10.064052 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.064034    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39895d50-c298-417f-ac88-7d693255237c-tls-cert\") pod \"authorino-5d58cf989c-47j9c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") " pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:10.071673 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.071650    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrklz\" (UniqueName: \"kubernetes.io/projected/39895d50-c298-417f-ac88-7d693255237c-kube-api-access-mrklz\") pod \"authorino-5d58cf989c-47j9c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") " pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:10.221847 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.221815    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:24:10.491662 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.491629    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d58cf989c-47j9c"]
Sep 10 17:24:10.494180 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:24:10.494143    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39895d50_c298_417f_ac88_7d693255237c.slice/crio-d405e32ae5e815c68caa012ea53e3eaea967279a610a13a77d727ed91e8e9e47 WatchSource:0}: Error finding container d405e32ae5e815c68caa012ea53e3eaea967279a610a13a77d727ed91e8e9e47: Status 404 returned error can't find the container with id d405e32ae5e815c68caa012ea53e3eaea967279a610a13a77d727ed91e8e9e47
Sep 10 17:24:10.845888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.845854    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d58cf989c-47j9c" event={"ID":"39895d50-c298-417f-ac88-7d693255237c","Type":"ContainerStarted","Data":"d405e32ae5e815c68caa012ea53e3eaea967279a610a13a77d727ed91e8e9e47"}
Sep 10 17:24:10.845888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.845874    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:10.854004 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.853981    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:10.868280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.868253    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsg5m\" (UniqueName: \"kubernetes.io/projected/40f6ee63-8acb-4550-ab6d-ec9843024bea-kube-api-access-xsg5m\") pod \"40f6ee63-8acb-4550-ab6d-ec9843024bea\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") "
Sep 10 17:24:10.868407 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.868347    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40f6ee63-8acb-4550-ab6d-ec9843024bea-tls-cert\") pod \"40f6ee63-8acb-4550-ab6d-ec9843024bea\" (UID: \"40f6ee63-8acb-4550-ab6d-ec9843024bea\") "
Sep 10 17:24:10.870582 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.870552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f6ee63-8acb-4550-ab6d-ec9843024bea-kube-api-access-xsg5m" (OuterVolumeSpecName: "kube-api-access-xsg5m") pod "40f6ee63-8acb-4550-ab6d-ec9843024bea" (UID: "40f6ee63-8acb-4550-ab6d-ec9843024bea"). InnerVolumeSpecName "kube-api-access-xsg5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:24:10.870702 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.870637    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f6ee63-8acb-4550-ab6d-ec9843024bea-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "40f6ee63-8acb-4550-ab6d-ec9843024bea" (UID: "40f6ee63-8acb-4550-ab6d-ec9843024bea"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:24:10.969145 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.969108    2554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40f6ee63-8acb-4550-ab6d-ec9843024bea-tls-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:24:10.969145 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:10.969141    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsg5m\" (UniqueName: \"kubernetes.io/projected/40f6ee63-8acb-4550-ab6d-ec9843024bea-kube-api-access-xsg5m\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:24:11.852429 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:11.852394    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q5t6b"
Sep 10 17:24:11.852429 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:11.852413    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d58cf989c-47j9c" event={"ID":"39895d50-c298-417f-ac88-7d693255237c","Type":"ContainerStarted","Data":"fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f"}
Sep 10 17:24:11.872502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:11.872435    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d58cf989c-47j9c" podStartSLOduration=2.395845631 podStartE2EDuration="2.872413873s" totalImagesPullingTime="476.568242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:11.870451283 +0000 UTC m=+647.226941684" watchObservedRunningTime="2026-09-10 17:24:11.872413873 +0000 UTC m=+647.228904269"
Sep 10 17:24:11.904028 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:11.903993    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q5t6b"]
Sep 10 17:24:11.911731 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:11.911699    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwhsm"]
Sep 10 17:24:11.911894 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:11.911874    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qwhsm" podUID="5ac020bb-ca18-4528-a830-8f2e0452d43e" containerName="authorino" containerID="cri-o://5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4" gracePeriod=30
Sep 10 17:24:11.913060 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:11.913034    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q5t6b"]
Sep 10 17:24:12.160867 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.160838    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwhsm"
Sep 10 17:24:12.178473 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.178439    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqps7\" (UniqueName: \"kubernetes.io/projected/5ac020bb-ca18-4528-a830-8f2e0452d43e-kube-api-access-wqps7\") pod \"5ac020bb-ca18-4528-a830-8f2e0452d43e\" (UID: \"5ac020bb-ca18-4528-a830-8f2e0452d43e\") "
Sep 10 17:24:12.180667 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.180641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac020bb-ca18-4528-a830-8f2e0452d43e-kube-api-access-wqps7" (OuterVolumeSpecName: "kube-api-access-wqps7") pod "5ac020bb-ca18-4528-a830-8f2e0452d43e" (UID: "5ac020bb-ca18-4528-a830-8f2e0452d43e"). InnerVolumeSpecName "kube-api-access-wqps7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:24:12.279790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.279754    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqps7\" (UniqueName: \"kubernetes.io/projected/5ac020bb-ca18-4528-a830-8f2e0452d43e-kube-api-access-wqps7\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:24:12.858063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.858026    2554 generic.go:356] "Generic (PLEG): container finished" podID="5ac020bb-ca18-4528-a830-8f2e0452d43e" containerID="5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4" exitCode=0
Sep 10 17:24:12.858479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.858070    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qwhsm"
Sep 10 17:24:12.858479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.858113    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwhsm" event={"ID":"5ac020bb-ca18-4528-a830-8f2e0452d43e","Type":"ContainerDied","Data":"5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4"}
Sep 10 17:24:12.858479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.858157    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qwhsm" event={"ID":"5ac020bb-ca18-4528-a830-8f2e0452d43e","Type":"ContainerDied","Data":"8e9e329b3259d34a098a140446cd362faa37d29dff116c8b2c3595cb53b4c0b9"}
Sep 10 17:24:12.858479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.858179    2554 scope.go:117] "RemoveContainer" containerID="5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4"
Sep 10 17:24:12.867161 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.867041    2554 scope.go:117] "RemoveContainer" containerID="5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4"
Sep 10 17:24:12.867470 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:24:12.867405    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4\": container with ID starting with 5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4 not found: ID does not exist" containerID="5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4"
Sep 10 17:24:12.867470 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.867440    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4"} err="failed to get container status \"5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4\": rpc error: code = NotFound desc = could not find container \"5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4\": container with ID starting with 5ff16d4a3433ec0ac69bd5ada50f3ca23b16759de3b7043035188ae1334548f4 not found: ID does not exist"
Sep 10 17:24:12.884384 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.884353    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwhsm"]
Sep 10 17:24:12.888131 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:12.888108    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qwhsm"]
Sep 10 17:24:13.114367 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:13.114291    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f6ee63-8acb-4550-ab6d-ec9843024bea" path="/var/lib/kubelet/pods/40f6ee63-8acb-4550-ab6d-ec9843024bea/volumes"
Sep 10 17:24:13.114517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:13.114505    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac020bb-ca18-4528-a830-8f2e0452d43e" path="/var/lib/kubelet/pods/5ac020bb-ca18-4528-a830-8f2e0452d43e/volumes"
Sep 10 17:24:55.726027 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:24:55.725989    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:25:14.282063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:14.282025    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:25:35.710170 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:35.710087    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:25:47.930673 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.930638    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"]
Sep 10 17:25:47.931077 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.931059    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac020bb-ca18-4528-a830-8f2e0452d43e" containerName="authorino"
Sep 10 17:25:47.931127 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.931080    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac020bb-ca18-4528-a830-8f2e0452d43e" containerName="authorino"
Sep 10 17:25:47.931164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.931145    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac020bb-ca18-4528-a830-8f2e0452d43e" containerName="authorino"
Sep 10 17:25:47.934079 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.934063    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:47.938974 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.938954    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 17:25:47.940011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.939989    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 17:25:47.940101 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.940024    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 17:25:47.941753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.941735    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sb45l\""
Sep 10 17:25:47.965717 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:47.965688    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"]
Sep 10 17:25:48.002656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.002631    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.002806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.002678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.002806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.002741    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.002806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.002799    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rrs\" (UniqueName: \"kubernetes.io/projected/0f74d224-0543-4664-8236-350bd39b007c-kube-api-access-w2rrs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.002959 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.002840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.002959 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.002883    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f74d224-0543-4664-8236-350bd39b007c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.103803 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.103770    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.103983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.103813    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.103983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.103852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rrs\" (UniqueName: \"kubernetes.io/projected/0f74d224-0543-4664-8236-350bd39b007c-kube-api-access-w2rrs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.103983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.103884    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.103983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.103918    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f74d224-0543-4664-8236-350bd39b007c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.103983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.103957    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.104232 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.104206    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.104293 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.104255    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.104394 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.104375    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.106083 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.106058    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f74d224-0543-4664-8236-350bd39b007c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.106465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.106443    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f74d224-0543-4664-8236-350bd39b007c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.113987 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.113963    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rrs\" (UniqueName: \"kubernetes.io/projected/0f74d224-0543-4664-8236-350bd39b007c-kube-api-access-w2rrs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf\" (UID: \"0f74d224-0543-4664-8236-350bd39b007c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.244184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.244097    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:48.522160 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.522128    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"]
Sep 10 17:25:48.523324 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:25:48.523303    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f74d224_0543_4664_8236_350bd39b007c.slice/crio-949458b5f248e193fe0d6e556d983219cffcd5760b256bdec1e61de2b756f0a2 WatchSource:0}: Error finding container 949458b5f248e193fe0d6e556d983219cffcd5760b256bdec1e61de2b756f0a2: Status 404 returned error can't find the container with id 949458b5f248e193fe0d6e556d983219cffcd5760b256bdec1e61de2b756f0a2
Sep 10 17:25:48.524656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.524637    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:25:48.885965 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:48.885926    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:25:49.263886 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:49.263789    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf" event={"ID":"0f74d224-0543-4664-8236-350bd39b007c","Type":"ContainerStarted","Data":"949458b5f248e193fe0d6e556d983219cffcd5760b256bdec1e61de2b756f0a2"}
Sep 10 17:25:50.269176 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:50.269139    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf" event={"ID":"0f74d224-0543-4664-8236-350bd39b007c","Type":"ContainerStarted","Data":"74f2841cfb7c9e4b8c776eb1cbf796f16ea7f545956eb3d392af5156ef72282c"}
Sep 10 17:25:50.293062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:50.293012    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf" podStartSLOduration=2.200773604 podStartE2EDuration="3.292996244s" totalImagesPullingTime="1.09222264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:50.291117561 +0000 UTC m=+745.647607961" watchObservedRunningTime="2026-09-10 17:25:50.292996244 +0000 UTC m=+745.649486644"
Sep 10 17:25:54.873498 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:54.873464    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:25:58.244734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:58.244679    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:58.245165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:58.244794    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:58.257492 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:58.257464    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:25:58.315215 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:25:58.315184    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf"
Sep 10 17:26:07.621587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.621555    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"]
Sep 10 17:26:07.625537 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.625505    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.628201 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.628180    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 17:26:07.637202 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.637179    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"]
Sep 10 17:26:07.683920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.683894    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.684063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.683940    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.684063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.683965    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.684063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.684019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c70b29-e439-4b97-947f-32f510cb8f8d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.684063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.684039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.684063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.684054    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5xg\" (UniqueName: \"kubernetes.io/projected/e4c70b29-e439-4b97-947f-32f510cb8f8d-kube-api-access-wj5xg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.784897 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.784861    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.784915    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.784947    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.784978    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c70b29-e439-4b97-947f-32f510cb8f8d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.785006    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.785031    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5xg\" (UniqueName: \"kubernetes.io/projected/e4c70b29-e439-4b97-947f-32f510cb8f8d-kube-api-access-wj5xg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.785378    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785461 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.785427    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.785515 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.785452    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.787195 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.787169    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c70b29-e439-4b97-947f-32f510cb8f8d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.787339 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.787322    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c70b29-e439-4b97-947f-32f510cb8f8d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.794815 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.794794    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5xg\" (UniqueName: \"kubernetes.io/projected/e4c70b29-e439-4b97-947f-32f510cb8f8d-kube-api-access-wj5xg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv\" (UID: \"e4c70b29-e439-4b97-947f-32f510cb8f8d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:07.935893 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:07.935799    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:08.207756 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:08.207732    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"]
Sep 10 17:26:08.210212 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:26:08.210186    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c70b29_e439_4b97_947f_32f510cb8f8d.slice/crio-e228e2574d99dce8f68332f4617c3dd32a9d7bc9725833f50bbababf731b450b WatchSource:0}: Error finding container e228e2574d99dce8f68332f4617c3dd32a9d7bc9725833f50bbababf731b450b: Status 404 returned error can't find the container with id e228e2574d99dce8f68332f4617c3dd32a9d7bc9725833f50bbababf731b450b
Sep 10 17:26:08.346266 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:08.346228    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv" event={"ID":"e4c70b29-e439-4b97-947f-32f510cb8f8d","Type":"ContainerStarted","Data":"e228e2574d99dce8f68332f4617c3dd32a9d7bc9725833f50bbababf731b450b"}
Sep 10 17:26:09.134714 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:09.134680    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:26:09.351613 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:09.351571    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv" event={"ID":"e4c70b29-e439-4b97-947f-32f510cb8f8d","Type":"ContainerStarted","Data":"5fb0c14534218d98903329075b1ded04c459a30057e9c04ccefb46d7cf46eb74"}
Sep 10 17:26:13.421803 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.421749    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv" podStartSLOduration=6.238612231 podStartE2EDuration="6.421732342s" totalImagesPullingTime="183.120111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:09.394136544 +0000 UTC m=+764.750626945" watchObservedRunningTime="2026-09-10 17:26:13.421732342 +0000 UTC m=+768.778222775"
Sep 10 17:26:13.423122 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.423105    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"]
Sep 10 17:26:13.425243 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.425228    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.428131 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.428107    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 17:26:13.446919 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.446892    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"]
Sep 10 17:26:13.536242 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.536205    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.536242 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.536243    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.536451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.536263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.536451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.536282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7hs\" (UniqueName: \"kubernetes.io/projected/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-kube-api-access-dp7hs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.536451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.536327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.536451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.536352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.636982 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.636951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637006    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637038    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7hs\" (UniqueName: \"kubernetes.io/projected/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-kube-api-access-dp7hs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637099    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637355 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637242    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637557 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637507    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637679 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.637679 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.637651    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.639328 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.639310    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.639618 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.639595    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.647715 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.647691    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7hs\" (UniqueName: \"kubernetes.io/projected/81c0372c-0cf3-4f2c-a725-1ffb7b83681e-kube-api-access-dp7hs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pwns8\" (UID: \"81c0372c-0cf3-4f2c-a725-1ffb7b83681e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:13.735316 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:13.735222    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:14.010782 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:14.010750    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"]
Sep 10 17:26:14.012256 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:26:14.012222    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c0372c_0cf3_4f2c_a725_1ffb7b83681e.slice/crio-876595c31f233c5c5b3c61caa05c670c4d931cf986804e852f8b5326621324cf WatchSource:0}: Error finding container 876595c31f233c5c5b3c61caa05c670c4d931cf986804e852f8b5326621324cf: Status 404 returned error can't find the container with id 876595c31f233c5c5b3c61caa05c670c4d931cf986804e852f8b5326621324cf
Sep 10 17:26:14.374612 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:14.374570    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8" event={"ID":"81c0372c-0cf3-4f2c-a725-1ffb7b83681e","Type":"ContainerStarted","Data":"9a4a5dbd3da34b2d057b6e5825f7e1d2b3137b6746d311dea5a5181077759721"}
Sep 10 17:26:14.374812 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:14.374628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8" event={"ID":"81c0372c-0cf3-4f2c-a725-1ffb7b83681e","Type":"ContainerStarted","Data":"876595c31f233c5c5b3c61caa05c670c4d931cf986804e852f8b5326621324cf"}
Sep 10 17:26:14.398911 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:14.398856    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8" podStartSLOduration=1.238409982 podStartE2EDuration="1.398840684s" totalImagesPullingTime="160.430702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:14.397644509 +0000 UTC m=+769.754134922" watchObservedRunningTime="2026-09-10 17:26:14.398840684 +0000 UTC m=+769.755331083"
Sep 10 17:26:15.288897 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:15.288862    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:26:17.936786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:17.936745    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:17.937262 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:17.936818    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:17.949514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:17.949483    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:18.403303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:18.403275    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv"
Sep 10 17:26:22.536454 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.536421    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"]
Sep 10 17:26:22.539682 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.539664    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.542648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.542630    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 17:26:22.556048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.556023    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"]
Sep 10 17:26:22.719301 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.719263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.719495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.719329    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.719495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.719382    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.719495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.719413    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7pp\" (UniqueName: \"kubernetes.io/projected/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-kube-api-access-nv7pp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.719495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.719453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.719495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.719474    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820071 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820036    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820079    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7pp\" (UniqueName: \"kubernetes.io/projected/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-kube-api-access-nv7pp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820108    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820243    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820309    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820581 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820560    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820724 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820694    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.820839 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.820748    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.822550 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.822507    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.822630 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.822614    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.830943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.830925    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7pp\" (UniqueName: \"kubernetes.io/projected/aaf3fb73-aa5f-4f97-81d4-fdefa589e9af-kube-api-access-nv7pp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s\" (UID: \"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:22.849000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:22.848971    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:23.124145 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:23.124118    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"]
Sep 10 17:26:23.126660 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:26:23.126631    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaf3fb73_aa5f_4f97_81d4_fdefa589e9af.slice/crio-81016cbc7cefeba6d6255a728f5fb2a142f0c08756de570991c9827c5849d298 WatchSource:0}: Error finding container 81016cbc7cefeba6d6255a728f5fb2a142f0c08756de570991c9827c5849d298: Status 404 returned error can't find the container with id 81016cbc7cefeba6d6255a728f5fb2a142f0c08756de570991c9827c5849d298
Sep 10 17:26:23.416016 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:23.415937    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s" event={"ID":"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af","Type":"ContainerStarted","Data":"866f17f58215152958418a9632ba165efd94298fd3b12ef9867339a944ddd85f"}
Sep 10 17:26:23.416016 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:23.415978    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s" event={"ID":"aaf3fb73-aa5f-4f97-81d4-fdefa589e9af","Type":"ContainerStarted","Data":"81016cbc7cefeba6d6255a728f5fb2a142f0c08756de570991c9827c5849d298"}
Sep 10 17:26:23.437768 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:23.437716    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s" podStartSLOduration=1.261369582 podStartE2EDuration="1.437699978s" totalImagesPullingTime="176.330396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:23.436559937 +0000 UTC m=+778.793050328" watchObservedRunningTime="2026-09-10 17:26:23.437699978 +0000 UTC m=+778.794190377"
Sep 10 17:26:23.735703 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:23.735609    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:23.735703 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:23.735675    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:23.748194 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:23.748164    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:24.431593 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:24.431568    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pwns8"
Sep 10 17:26:24.498497 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:24.498466    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:26:32.849741 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:32.849700    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:32.850130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:32.849758    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:32.861862 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:32.861841    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:26:33.470200 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:26:33.470169    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s"
Sep 10 17:27:27.269948 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.269913    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-775ccfbcbd-w4z5r"]
Sep 10 17:27:27.272758 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.272742    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.285104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.285077    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775ccfbcbd-w4z5r"]
Sep 10 17:27:27.389667 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.389633    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4789de8-e863-4760-8ecd-06118e8b484f-tls-cert\") pod \"authorino-775ccfbcbd-w4z5r\" (UID: \"c4789de8-e863-4760-8ecd-06118e8b484f\") " pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.389823 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.389694    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kcn\" (UniqueName: \"kubernetes.io/projected/c4789de8-e863-4760-8ecd-06118e8b484f-kube-api-access-44kcn\") pod \"authorino-775ccfbcbd-w4z5r\" (UID: \"c4789de8-e863-4760-8ecd-06118e8b484f\") " pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.490235 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.490201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4789de8-e863-4760-8ecd-06118e8b484f-tls-cert\") pod \"authorino-775ccfbcbd-w4z5r\" (UID: \"c4789de8-e863-4760-8ecd-06118e8b484f\") " pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.490394 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.490268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kcn\" (UniqueName: \"kubernetes.io/projected/c4789de8-e863-4760-8ecd-06118e8b484f-kube-api-access-44kcn\") pod \"authorino-775ccfbcbd-w4z5r\" (UID: \"c4789de8-e863-4760-8ecd-06118e8b484f\") " pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.492639 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.492611    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4789de8-e863-4760-8ecd-06118e8b484f-tls-cert\") pod \"authorino-775ccfbcbd-w4z5r\" (UID: \"c4789de8-e863-4760-8ecd-06118e8b484f\") " pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.499649 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.499619    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kcn\" (UniqueName: \"kubernetes.io/projected/c4789de8-e863-4760-8ecd-06118e8b484f-kube-api-access-44kcn\") pod \"authorino-775ccfbcbd-w4z5r\" (UID: \"c4789de8-e863-4760-8ecd-06118e8b484f\") " pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.583726 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.583695    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775ccfbcbd-w4z5r"
Sep 10 17:27:27.864360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:27.864338    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775ccfbcbd-w4z5r"]
Sep 10 17:27:27.866217 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:27:27.866190    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4789de8_e863_4760_8ecd_06118e8b484f.slice/crio-2f817392e75378f4397ac028f18cda3cfe26ddb2330718a12d8e053f4a23a96d WatchSource:0}: Error finding container 2f817392e75378f4397ac028f18cda3cfe26ddb2330718a12d8e053f4a23a96d: Status 404 returned error can't find the container with id 2f817392e75378f4397ac028f18cda3cfe26ddb2330718a12d8e053f4a23a96d
Sep 10 17:27:28.707826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:28.707790    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775ccfbcbd-w4z5r" event={"ID":"c4789de8-e863-4760-8ecd-06118e8b484f","Type":"ContainerStarted","Data":"2f817392e75378f4397ac028f18cda3cfe26ddb2330718a12d8e053f4a23a96d"}
Sep 10 17:27:29.716310 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:29.715560    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775ccfbcbd-w4z5r" event={"ID":"c4789de8-e863-4760-8ecd-06118e8b484f","Type":"ContainerStarted","Data":"c05204e0ed96f4924174ac160501abebb1b3c3b1a23b640ecfbcc95ab1df068a"}
Sep 10 17:27:29.738595 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:29.738515    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-775ccfbcbd-w4z5r" podStartSLOduration=1.774645922 podStartE2EDuration="2.738480439s" totalImagesPullingTime="963.834517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:29.736815392 +0000 UTC m=+845.093305793" watchObservedRunningTime="2026-09-10 17:27:29.738480439 +0000 UTC m=+845.094970841"
Sep 10 17:27:29.791489 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:29.791446    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d58cf989c-47j9c"]
Sep 10 17:27:29.791758 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:29.791728    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d58cf989c-47j9c" podUID="39895d50-c298-417f-ac88-7d693255237c" containerName="authorino" containerID="cri-o://fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f" gracePeriod=30
Sep 10 17:27:30.057590 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.057550    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:27:30.115123 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.115092    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39895d50-c298-417f-ac88-7d693255237c-tls-cert\") pod \"39895d50-c298-417f-ac88-7d693255237c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") "
Sep 10 17:27:30.115312 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.115142    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrklz\" (UniqueName: \"kubernetes.io/projected/39895d50-c298-417f-ac88-7d693255237c-kube-api-access-mrklz\") pod \"39895d50-c298-417f-ac88-7d693255237c\" (UID: \"39895d50-c298-417f-ac88-7d693255237c\") "
Sep 10 17:27:30.117208 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.117163    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39895d50-c298-417f-ac88-7d693255237c-kube-api-access-mrklz" (OuterVolumeSpecName: "kube-api-access-mrklz") pod "39895d50-c298-417f-ac88-7d693255237c" (UID: "39895d50-c298-417f-ac88-7d693255237c"). InnerVolumeSpecName "kube-api-access-mrklz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:30.124983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.124950    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39895d50-c298-417f-ac88-7d693255237c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "39895d50-c298-417f-ac88-7d693255237c" (UID: "39895d50-c298-417f-ac88-7d693255237c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:30.216878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.216831    2554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39895d50-c298-417f-ac88-7d693255237c-tls-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:27:30.216878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.216879    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrklz\" (UniqueName: \"kubernetes.io/projected/39895d50-c298-417f-ac88-7d693255237c-kube-api-access-mrklz\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:27:30.720997 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.720958    2554 generic.go:356] "Generic (PLEG): container finished" podID="39895d50-c298-417f-ac88-7d693255237c" containerID="fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f" exitCode=0
Sep 10 17:27:30.721448 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.721015    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d58cf989c-47j9c"
Sep 10 17:27:30.721448 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.721006    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d58cf989c-47j9c" event={"ID":"39895d50-c298-417f-ac88-7d693255237c","Type":"ContainerDied","Data":"fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f"}
Sep 10 17:27:30.721448 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.721138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d58cf989c-47j9c" event={"ID":"39895d50-c298-417f-ac88-7d693255237c","Type":"ContainerDied","Data":"d405e32ae5e815c68caa012ea53e3eaea967279a610a13a77d727ed91e8e9e47"}
Sep 10 17:27:30.721448 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.721177    2554 scope.go:117] "RemoveContainer" containerID="fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f"
Sep 10 17:27:30.730863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.730835    2554 scope.go:117] "RemoveContainer" containerID="fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f"
Sep 10 17:27:30.731253 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:27:30.731224    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f\": container with ID starting with fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f not found: ID does not exist" containerID="fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f"
Sep 10 17:27:30.731330 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.731266    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f"} err="failed to get container status \"fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f\": rpc error: code = NotFound desc = could not find container \"fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f\": container with ID starting with fca184e67bbb7d7f39b21d623122ea898de1a2612b3e16159c8e3e7f7db9400f not found: ID does not exist"
Sep 10 17:27:30.748383 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.748354    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d58cf989c-47j9c"]
Sep 10 17:27:30.758544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:30.758504    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d58cf989c-47j9c"]
Sep 10 17:27:31.114490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:31.114455    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39895d50-c298-417f-ac88-7d693255237c" path="/var/lib/kubelet/pods/39895d50-c298-417f-ac88-7d693255237c/volumes"
Sep 10 17:27:47.856954 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:47.856920    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:27:58.705042 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:58.705004    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:27:59.555618 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.555582    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"]
Sep 10 17:27:59.555937 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.555925    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39895d50-c298-417f-ac88-7d693255237c" containerName="authorino"
Sep 10 17:27:59.555984 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.555939    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="39895d50-c298-417f-ac88-7d693255237c" containerName="authorino"
Sep 10 17:27:59.556018 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.556000    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="39895d50-c298-417f-ac88-7d693255237c" containerName="authorino"
Sep 10 17:27:59.558765 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.558746    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:27:59.561840 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.561820    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e52f3775-dockercfg-dnlpk\""
Sep 10 17:27:59.562021 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.562009    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e52f3775\""
Sep 10 17:27:59.573037 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.573016    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"]
Sep 10 17:27:59.685543 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.685487    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgp95\" (UniqueName: \"kubernetes.io/projected/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-kube-api-access-cgp95\") pod \"payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:27:59.685699 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.685615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:27:59.786337 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.786299    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:27:59.786785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.786459    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgp95\" (UniqueName: \"kubernetes.io/projected/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-kube-api-access-cgp95\") pod \"payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:27:59.787003 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.786981    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:27:59.798041 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.798018    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgp95\" (UniqueName: \"kubernetes.io/projected/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-kube-api-access-cgp95\") pod \"payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:27:59.868588 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:27:59.868506    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:28:00.143438 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:00.143396    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"]
Sep 10 17:28:00.145749 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:28:00.145716    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec819c3_9dbb_4707_81df_d998b7cf8eb9.slice/crio-e73d36aeab8667d9263b8df81a658b0295544b7764118119a136edb7a6470e83 WatchSource:0}: Error finding container e73d36aeab8667d9263b8df81a658b0295544b7764118119a136edb7a6470e83: Status 404 returned error can't find the container with id e73d36aeab8667d9263b8df81a658b0295544b7764118119a136edb7a6470e83
Sep 10 17:28:00.858501 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:00.858426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq" event={"ID":"6ec819c3-9dbb-4707-81df-d998b7cf8eb9","Type":"ContainerStarted","Data":"e73d36aeab8667d9263b8df81a658b0295544b7764118119a136edb7a6470e83"}
Sep 10 17:28:01.197198 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.197116    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"]
Sep 10 17:28:01.200672 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.200648    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.206670 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.206622    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-c38922a9-kserve-self-signed-certs\""
Sep 10 17:28:01.214110 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.214086    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"]
Sep 10 17:28:01.299033 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.298997    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba63699-d7b5-4fff-ade7-1afcc4d62684-tls-certs\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.299206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.299062    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-tmp-dir\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.299206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.299091    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-dshm\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.299206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.299109    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-model-cache\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.299206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.299170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-home\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.299206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.299201    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkmw\" (UniqueName: \"kubernetes.io/projected/fba63699-d7b5-4fff-ade7-1afcc4d62684-kube-api-access-mtkmw\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400075 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba63699-d7b5-4fff-ade7-1afcc4d62684-tls-certs\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400265 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400111    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-tmp-dir\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400265 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400140    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-dshm\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400265 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-model-cache\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400265 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400198    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-home\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400265 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400237    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkmw\" (UniqueName: \"kubernetes.io/projected/fba63699-d7b5-4fff-ade7-1afcc4d62684-kube-api-access-mtkmw\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400607 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400561    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-tmp-dir\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400805 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400780    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-home\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.400887 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.400859    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-model-cache\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.402862 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.402831    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-dshm\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.403079 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.403032    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba63699-d7b5-4fff-ade7-1afcc4d62684-tls-certs\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.409288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.409265    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkmw\" (UniqueName: \"kubernetes.io/projected/fba63699-d7b5-4fff-ade7-1afcc4d62684-kube-api-access-mtkmw\") pod \"e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") " pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:01.513744 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:01.513667    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:02.507232 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:02.507203    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"]
Sep 10 17:28:02.509829 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:28:02.509802    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfba63699_d7b5_4fff_ade7_1afcc4d62684.slice/crio-3d501933092e04710ecdca3abbaa3420a0b6e3c09151edd9b4b66da61130c8db WatchSource:0}: Error finding container 3d501933092e04710ecdca3abbaa3420a0b6e3c09151edd9b4b66da61130c8db: Status 404 returned error can't find the container with id 3d501933092e04710ecdca3abbaa3420a0b6e3c09151edd9b4b66da61130c8db
Sep 10 17:28:02.878237 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:02.878198    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks" event={"ID":"fba63699-d7b5-4fff-ade7-1afcc4d62684","Type":"ContainerStarted","Data":"00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab"}
Sep 10 17:28:02.878420 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:02.878252    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks" event={"ID":"fba63699-d7b5-4fff-ade7-1afcc4d62684","Type":"ContainerStarted","Data":"3d501933092e04710ecdca3abbaa3420a0b6e3c09151edd9b4b66da61130c8db"}
Sep 10 17:28:02.879768 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:02.879737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq" event={"ID":"6ec819c3-9dbb-4707-81df-d998b7cf8eb9","Type":"ContainerStarted","Data":"2e48c69164b0b81b6f3d8bd9c5a2db76cff64009e5e6ff23cd3725d31b2319f5"}
Sep 10 17:28:02.879873 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:02.879834    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:28:02.904998 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:02.904954    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks" podStartSLOduration=1.904938875 podStartE2EDuration="1.904938875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:02.903393028 +0000 UTC m=+878.259883428" watchObservedRunningTime="2026-09-10 17:28:02.904938875 +0000 UTC m=+878.261429317"
Sep 10 17:28:02.927398 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:02.927346    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq" podStartSLOduration=1.779093321 podStartE2EDuration="3.92732885s" totalImagesPullingTime="2.148235529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:02.926049971 +0000 UTC m=+878.282540376" watchObservedRunningTime="2026-09-10 17:28:02.92732885 +0000 UTC m=+878.283819249"
Sep 10 17:28:11.514707 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.514669    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:11.515073 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.514722    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:11.527191 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.527166    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:11.892794 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.892753    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"]
Sep 10 17:28:11.896634 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.896612    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"
Sep 10 17:28:11.900128 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.900104    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:28:11.900966 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.900948    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:28:11.908740 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.908716    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"]
Sep 10 17:28:11.935721 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.935691    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:28:11.999013 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:11.998984    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e13ad-3500-427f-a043-c1840126d168-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm\" (UID: \"dc2e13ad-3500-427f-a043-c1840126d168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"
Sep 10 17:28:12.100467 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.100424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e13ad-3500-427f-a043-c1840126d168-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm\" (UID: \"dc2e13ad-3500-427f-a043-c1840126d168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"
Sep 10 17:28:12.101062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.101043    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e13ad-3500-427f-a043-c1840126d168-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm\" (UID: \"dc2e13ad-3500-427f-a043-c1840126d168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"
Sep 10 17:28:12.235089 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.235013    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"
Sep 10 17:28:12.528432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.528350    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"]
Sep 10 17:28:12.531233 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:28:12.531207    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2e13ad_3500_427f_a043_c1840126d168.slice/crio-8c5dd0ab22d6ead66c6dcc58ec5c22892d509a5a875e6c08d78b7b9097857876 WatchSource:0}: Error finding container 8c5dd0ab22d6ead66c6dcc58ec5c22892d509a5a875e6c08d78b7b9097857876: Status 404 returned error can't find the container with id 8c5dd0ab22d6ead66c6dcc58ec5c22892d509a5a875e6c08d78b7b9097857876
Sep 10 17:28:12.732324 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.732290    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"]
Sep 10 17:28:12.737317 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.737296    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.743671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.743634    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-15ece2-serving-cert\""
Sep 10 17:28:12.743831 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.743700    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 17:28:12.743831 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.743651    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2hfkp\""
Sep 10 17:28:12.745514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.745492    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 17:28:12.758123 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.758094    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"]
Sep 10 17:28:12.807278 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.807251    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.807444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.807285    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-metrics-certs\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.807444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.807408    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddde51-c250-49d2-a931-b43a53956de2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.807523 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.807492    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhbrv\" (UniqueName: \"kubernetes.io/projected/68ddde51-c250-49d2-a931-b43a53956de2-kube-api-access-zhbrv\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.908362 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.908331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhbrv\" (UniqueName: \"kubernetes.io/projected/68ddde51-c250-49d2-a931-b43a53956de2-kube-api-access-zhbrv\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.908514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.908372    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.908514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.908392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-metrics-certs\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.908514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.908455    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddde51-c250-49d2-a931-b43a53956de2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.908514 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:28:12.908494    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-serving-cert: secret "maas-api-e2e-inf-b-15ece2-serving-cert" not found
Sep 10 17:28:12.908683 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:28:12.908600    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls podName:68ddde51-c250-49d2-a931-b43a53956de2 nodeName:}" failed. No retries permitted until 2026-09-10 17:28:13.408576485 +0000 UTC m=+888.765066866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls") pod "maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q" (UID: "68ddde51-c250-49d2-a931-b43a53956de2") : secret "maas-api-e2e-inf-b-15ece2-serving-cert" not found
Sep 10 17:28:12.909126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.909108    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddde51-c250-49d2-a931-b43a53956de2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.910928 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.910904    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-metrics-certs\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.918276 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.918253    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhbrv\" (UniqueName: \"kubernetes.io/projected/68ddde51-c250-49d2-a931-b43a53956de2-kube-api-access-zhbrv\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:12.928399 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:12.928374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm" event={"ID":"dc2e13ad-3500-427f-a043-c1840126d168","Type":"ContainerStarted","Data":"8c5dd0ab22d6ead66c6dcc58ec5c22892d509a5a875e6c08d78b7b9097857876"}
Sep 10 17:28:13.413480 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:13.413443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:13.416004 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:13.415980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls\") pod \"maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:13.648706 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:13.648662    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:13.886659 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:13.886624    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:28:13.938367 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:13.938293    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"]
Sep 10 17:28:13.943054 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:28:13.943027    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ddde51_c250_49d2_a931_b43a53956de2.slice/crio-f19a5d1fc022ab2f13a4d30c346bf5e725fb619e08f615dbbe8003a093726b41 WatchSource:0}: Error finding container f19a5d1fc022ab2f13a4d30c346bf5e725fb619e08f615dbbe8003a093726b41: Status 404 returned error can't find the container with id f19a5d1fc022ab2f13a4d30c346bf5e725fb619e08f615dbbe8003a093726b41
Sep 10 17:28:14.942674 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:14.942635    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q" event={"ID":"68ddde51-c250-49d2-a931-b43a53956de2","Type":"ContainerStarted","Data":"f19a5d1fc022ab2f13a4d30c346bf5e725fb619e08f615dbbe8003a093726b41"}
Sep 10 17:28:15.949553 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:15.949448    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm" event={"ID":"dc2e13ad-3500-427f-a043-c1840126d168","Type":"ContainerStarted","Data":"5b82c34e8b96bee1be852977b7e574d5c6cb0dba609ae33edc36403d7ce96f40"}
Sep 10 17:28:15.988756 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:15.988702    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm" podStartSLOduration=2.747593637 podStartE2EDuration="4.988679137s" totalImagesPullingTime="2.2410855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:15.988046496 +0000 UTC m=+891.344536897" watchObservedRunningTime="2026-09-10 17:28:15.988679137 +0000 UTC m=+891.345169541"
Sep 10 17:28:16.955408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:16.955300    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q" event={"ID":"68ddde51-c250-49d2-a931-b43a53956de2","Type":"ContainerStarted","Data":"887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f"}
Sep 10 17:28:16.955408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:16.955364    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:16.957063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:16.957027    2554 generic.go:356] "Generic (PLEG): container finished" podID="dc2e13ad-3500-427f-a043-c1840126d168" containerID="5b82c34e8b96bee1be852977b7e574d5c6cb0dba609ae33edc36403d7ce96f40" exitCode=0
Sep 10 17:28:16.957165 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:16.957062    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm" event={"ID":"dc2e13ad-3500-427f-a043-c1840126d168","Type":"ContainerDied","Data":"5b82c34e8b96bee1be852977b7e574d5c6cb0dba609ae33edc36403d7ce96f40"}
Sep 10 17:28:16.984589 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:16.982967    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q" podStartSLOduration=2.325798206 podStartE2EDuration="4.98294287s" totalImagesPullingTime="2.657144664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:16.980953917 +0000 UTC m=+892.337444318" watchObservedRunningTime="2026-09-10 17:28:16.98294287 +0000 UTC m=+892.339433272"
Sep 10 17:28:18.093235 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.093204    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"
Sep 10 17:28:18.163377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.163345    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e13ad-3500-427f-a043-c1840126d168-maas-api-service-ca\") pod \"dc2e13ad-3500-427f-a043-c1840126d168\" (UID: \"dc2e13ad-3500-427f-a043-c1840126d168\") "
Sep 10 17:28:18.163781 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.163745    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2e13ad-3500-427f-a043-c1840126d168-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc2e13ad-3500-427f-a043-c1840126d168" (UID: "dc2e13ad-3500-427f-a043-c1840126d168"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:28:18.264977 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.264901    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc2e13ad-3500-427f-a043-c1840126d168-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:28:18.420934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.420896    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:28:18.968803 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.968771    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"
Sep 10 17:28:18.968946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.968769    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm" event={"ID":"dc2e13ad-3500-427f-a043-c1840126d168","Type":"ContainerDied","Data":"8c5dd0ab22d6ead66c6dcc58ec5c22892d509a5a875e6c08d78b7b9097857876"}
Sep 10 17:28:18.968946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:18.968890    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5dd0ab22d6ead66c6dcc58ec5c22892d509a5a875e6c08d78b7b9097857876"
Sep 10 17:28:22.831524 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:22.831483    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"]
Sep 10 17:28:22.831931 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:22.831762    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq" podUID="6ec819c3-9dbb-4707-81df-d998b7cf8eb9" containerName="payload-pre-processing" containerID="cri-o://2e48c69164b0b81b6f3d8bd9c5a2db76cff64009e5e6ff23cd3725d31b2319f5" gracePeriod=30
Sep 10 17:28:22.968082 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:22.968053    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:28:22.987997 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:22.987960    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ec819c3-9dbb-4707-81df-d998b7cf8eb9" containerID="2e48c69164b0b81b6f3d8bd9c5a2db76cff64009e5e6ff23cd3725d31b2319f5" exitCode=0
Sep 10 17:28:22.988137 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:22.988034    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq" event={"ID":"6ec819c3-9dbb-4707-81df-d998b7cf8eb9","Type":"ContainerDied","Data":"2e48c69164b0b81b6f3d8bd9c5a2db76cff64009e5e6ff23cd3725d31b2319f5"}
Sep 10 17:28:23.055380 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.055338    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"]
Sep 10 17:28:23.058578 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.058551    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e52f3775-g8tfm"]
Sep 10 17:28:23.093456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.093399    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:28:23.114687 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.114655    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc2e13ad-3500-427f-a043-c1840126d168" path="/var/lib/kubelet/pods/dc2e13ad-3500-427f-a043-c1840126d168/volumes"
Sep 10 17:28:23.209939 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.209910    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-plugins-config-volume\") pod \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") "
Sep 10 17:28:23.210116 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.209982    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgp95\" (UniqueName: \"kubernetes.io/projected/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-kube-api-access-cgp95\") pod \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\" (UID: \"6ec819c3-9dbb-4707-81df-d998b7cf8eb9\") "
Sep 10 17:28:23.210354 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.210326    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ec819c3-9dbb-4707-81df-d998b7cf8eb9" (UID: "6ec819c3-9dbb-4707-81df-d998b7cf8eb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:28:23.212196 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.212180    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-kube-api-access-cgp95" (OuterVolumeSpecName: "kube-api-access-cgp95") pod "6ec819c3-9dbb-4707-81df-d998b7cf8eb9" (UID: "6ec819c3-9dbb-4707-81df-d998b7cf8eb9"). InnerVolumeSpecName "kube-api-access-cgp95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:28:23.310675 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.310644    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:28:23.310675 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.310672    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgp95\" (UniqueName: \"kubernetes.io/projected/6ec819c3-9dbb-4707-81df-d998b7cf8eb9-kube-api-access-cgp95\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:28:23.994384 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.994339    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq" event={"ID":"6ec819c3-9dbb-4707-81df-d998b7cf8eb9","Type":"ContainerDied","Data":"e73d36aeab8667d9263b8df81a658b0295544b7764118119a136edb7a6470e83"}
Sep 10 17:28:23.994785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.994408    2554 scope.go:117] "RemoveContainer" containerID="2e48c69164b0b81b6f3d8bd9c5a2db76cff64009e5e6ff23cd3725d31b2319f5"
Sep 10 17:28:23.994785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:23.994361    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"
Sep 10 17:28:24.048244 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:24.048211    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"]
Sep 10 17:28:24.066803 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:24.066776    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e52f3775-56b7b8c49d-qhjjq"]
Sep 10 17:28:25.050936 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:25.050905    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:28:25.051359 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:25.051115    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:28:25.114087 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:25.114055    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec819c3-9dbb-4707-81df-d998b7cf8eb9" path="/var/lib/kubelet/pods/6ec819c3-9dbb-4707-81df-d998b7cf8eb9/volumes"
Sep 10 17:28:31.147517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.147480    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"]
Sep 10 17:28:31.148005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.147840    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec819c3-9dbb-4707-81df-d998b7cf8eb9" containerName="payload-pre-processing"
Sep 10 17:28:31.148005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.147853    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec819c3-9dbb-4707-81df-d998b7cf8eb9" containerName="payload-pre-processing"
Sep 10 17:28:31.148005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.147879    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2e13ad-3500-427f-a043-c1840126d168" containerName="revoke-keys"
Sep 10 17:28:31.148005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.147884    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2e13ad-3500-427f-a043-c1840126d168" containerName="revoke-keys"
Sep 10 17:28:31.148005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.147937    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc2e13ad-3500-427f-a043-c1840126d168" containerName="revoke-keys"
Sep 10 17:28:31.148005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.147948    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec819c3-9dbb-4707-81df-d998b7cf8eb9" containerName="payload-pre-processing"
Sep 10 17:28:31.153878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.153861    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.158889 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.158859    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2d96ccec-dockercfg-d4xcj\""
Sep 10 17:28:31.159170 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.159146    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2d96ccec\""
Sep 10 17:28:31.162360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.162338    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"]
Sep 10 17:28:31.283012 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.282970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c9ca5-734c-4efd-85ee-853b942031cc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.283188 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.283042    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7r6\" (UniqueName: \"kubernetes.io/projected/533c9ca5-734c-4efd-85ee-853b942031cc-kube-api-access-vt7r6\") pod \"payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.384353 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.384316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7r6\" (UniqueName: \"kubernetes.io/projected/533c9ca5-734c-4efd-85ee-853b942031cc-kube-api-access-vt7r6\") pod \"payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.384643 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.384436    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c9ca5-734c-4efd-85ee-853b942031cc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.385175 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.385150    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c9ca5-734c-4efd-85ee-853b942031cc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.394837 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.394817    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7r6\" (UniqueName: \"kubernetes.io/projected/533c9ca5-734c-4efd-85ee-853b942031cc-kube-api-access-vt7r6\") pod \"payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.464286 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.464204    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:31.783958 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:31.783693    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"]
Sep 10 17:28:31.786080 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:28:31.786052    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533c9ca5_734c_4efd_85ee_853b942031cc.slice/crio-da9ea9b19917b95e04217dc4946aaeadcb057d2ed623c52da5c6bd76664f8bd7 WatchSource:0}: Error finding container da9ea9b19917b95e04217dc4946aaeadcb057d2ed623c52da5c6bd76664f8bd7: Status 404 returned error can't find the container with id da9ea9b19917b95e04217dc4946aaeadcb057d2ed623c52da5c6bd76664f8bd7
Sep 10 17:28:32.039747 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:32.039659    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26" event={"ID":"533c9ca5-734c-4efd-85ee-853b942031cc","Type":"ContainerStarted","Data":"faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386"}
Sep 10 17:28:32.039747 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:32.039714    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26" event={"ID":"533c9ca5-734c-4efd-85ee-853b942031cc","Type":"ContainerStarted","Data":"da9ea9b19917b95e04217dc4946aaeadcb057d2ed623c52da5c6bd76664f8bd7"}
Sep 10 17:28:32.039938 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:32.039786    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:32.059495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:32.059437    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26" podStartSLOduration=1.059414668 podStartE2EDuration="1.059414668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:32.058475016 +0000 UTC m=+907.414965417" watchObservedRunningTime="2026-09-10 17:28:32.059414668 +0000 UTC m=+907.415905074"
Sep 10 17:28:38.749642 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:38.749606    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:28:43.047706 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:43.047672    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:28:57.265498 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.265465    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"]
Sep 10 17:28:57.267835 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.267818    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"
Sep 10 17:28:57.271762 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.271740    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:28:57.283417 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.283392    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"]
Sep 10 17:28:57.329444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.329417    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525c83d8-1907-425d-8cfc-a113435af06f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r\" (UID: \"525c83d8-1907-425d-8cfc-a113435af06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"
Sep 10 17:28:57.430378 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.430329    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525c83d8-1907-425d-8cfc-a113435af06f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r\" (UID: \"525c83d8-1907-425d-8cfc-a113435af06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"
Sep 10 17:28:57.430946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.430923    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525c83d8-1907-425d-8cfc-a113435af06f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r\" (UID: \"525c83d8-1907-425d-8cfc-a113435af06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"
Sep 10 17:28:57.577801 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.577768    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"
Sep 10 17:28:57.854767 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:57.854737    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"]
Sep 10 17:28:57.856177 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:28:57.856151    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525c83d8_1907_425d_8cfc_a113435af06f.slice/crio-ffcd2982893fb5d17503971074ccf072ccf3458c7057894f48f83b5604f282fa WatchSource:0}: Error finding container ffcd2982893fb5d17503971074ccf072ccf3458c7057894f48f83b5604f282fa: Status 404 returned error can't find the container with id ffcd2982893fb5d17503971074ccf072ccf3458c7057894f48f83b5604f282fa
Sep 10 17:28:58.162884 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:58.162858    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r" event={"ID":"525c83d8-1907-425d-8cfc-a113435af06f","Type":"ContainerStarted","Data":"419177e01eac6ba827d75485057671c90f97fff94a30eeb0a9f04c77340aed3e"}
Sep 10 17:28:58.162999 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:58.162912    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r" event={"ID":"525c83d8-1907-425d-8cfc-a113435af06f","Type":"ContainerStarted","Data":"ffcd2982893fb5d17503971074ccf072ccf3458c7057894f48f83b5604f282fa"}
Sep 10 17:28:58.188030 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:58.187984    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r" podStartSLOduration=1.187969314 podStartE2EDuration="1.187969314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:58.187678035 +0000 UTC m=+933.544168432" watchObservedRunningTime="2026-09-10 17:28:58.187969314 +0000 UTC m=+933.544459714"
Sep 10 17:28:59.169086 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:59.169049    2554 generic.go:356] "Generic (PLEG): container finished" podID="525c83d8-1907-425d-8cfc-a113435af06f" containerID="419177e01eac6ba827d75485057671c90f97fff94a30eeb0a9f04c77340aed3e" exitCode=0
Sep 10 17:28:59.169453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:28:59.169111    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r" event={"ID":"525c83d8-1907-425d-8cfc-a113435af06f","Type":"ContainerDied","Data":"419177e01eac6ba827d75485057671c90f97fff94a30eeb0a9f04c77340aed3e"}
Sep 10 17:29:00.298256 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:00.298234    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"
Sep 10 17:29:00.357273 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:00.357237    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525c83d8-1907-425d-8cfc-a113435af06f-maas-api-service-ca\") pod \"525c83d8-1907-425d-8cfc-a113435af06f\" (UID: \"525c83d8-1907-425d-8cfc-a113435af06f\") "
Sep 10 17:29:00.357614 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:00.357591    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525c83d8-1907-425d-8cfc-a113435af06f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "525c83d8-1907-425d-8cfc-a113435af06f" (UID: "525c83d8-1907-425d-8cfc-a113435af06f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:00.458900 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:00.458807    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525c83d8-1907-425d-8cfc-a113435af06f-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:01.178962 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:01.178931    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"
Sep 10 17:29:01.179099 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:01.178962    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r" event={"ID":"525c83d8-1907-425d-8cfc-a113435af06f","Type":"ContainerDied","Data":"ffcd2982893fb5d17503971074ccf072ccf3458c7057894f48f83b5604f282fa"}
Sep 10 17:29:01.179099 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:01.179012    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffcd2982893fb5d17503971074ccf072ccf3458c7057894f48f83b5604f282fa"
Sep 10 17:29:05.499428 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:05.499397    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:29:09.677926 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:09.677893    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"]
Sep 10 17:29:09.678329 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:09.678143    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26" podUID="533c9ca5-734c-4efd-85ee-853b942031cc" containerName="payload-processing" containerID="cri-o://faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386" gracePeriod=30
Sep 10 17:29:09.927854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:09.927829    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:29:10.050206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.050181    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c9ca5-734c-4efd-85ee-853b942031cc-plugins-config-volume\") pod \"533c9ca5-734c-4efd-85ee-853b942031cc\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") "
Sep 10 17:29:10.050390 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.050284    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt7r6\" (UniqueName: \"kubernetes.io/projected/533c9ca5-734c-4efd-85ee-853b942031cc-kube-api-access-vt7r6\") pod \"533c9ca5-734c-4efd-85ee-853b942031cc\" (UID: \"533c9ca5-734c-4efd-85ee-853b942031cc\") "
Sep 10 17:29:10.052201 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.051229    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533c9ca5-734c-4efd-85ee-853b942031cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "533c9ca5-734c-4efd-85ee-853b942031cc" (UID: "533c9ca5-734c-4efd-85ee-853b942031cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:10.053437 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.053409    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533c9ca5-734c-4efd-85ee-853b942031cc-kube-api-access-vt7r6" (OuterVolumeSpecName: "kube-api-access-vt7r6") pod "533c9ca5-734c-4efd-85ee-853b942031cc" (UID: "533c9ca5-734c-4efd-85ee-853b942031cc"). InnerVolumeSpecName "kube-api-access-vt7r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:10.106387 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.106356    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"]
Sep 10 17:29:10.113474 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.113452    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d96ccec-ths8r"]
Sep 10 17:29:10.151254 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.151221    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt7r6\" (UniqueName: \"kubernetes.io/projected/533c9ca5-734c-4efd-85ee-853b942031cc-kube-api-access-vt7r6\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:10.151254 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.151249    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533c9ca5-734c-4efd-85ee-853b942031cc-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:10.218640 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.218612    2554 generic.go:356] "Generic (PLEG): container finished" podID="533c9ca5-734c-4efd-85ee-853b942031cc" containerID="faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386" exitCode=0
Sep 10 17:29:10.218765 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.218656    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"
Sep 10 17:29:10.218765 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.218701    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26" event={"ID":"533c9ca5-734c-4efd-85ee-853b942031cc","Type":"ContainerDied","Data":"faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386"}
Sep 10 17:29:10.218765 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.218756    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26" event={"ID":"533c9ca5-734c-4efd-85ee-853b942031cc","Type":"ContainerDied","Data":"da9ea9b19917b95e04217dc4946aaeadcb057d2ed623c52da5c6bd76664f8bd7"}
Sep 10 17:29:10.218921 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.218786    2554 scope.go:117] "RemoveContainer" containerID="faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386"
Sep 10 17:29:10.227798 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.227784    2554 scope.go:117] "RemoveContainer" containerID="faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386"
Sep 10 17:29:10.228076 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:29:10.228057    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386\": container with ID starting with faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386 not found: ID does not exist" containerID="faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386"
Sep 10 17:29:10.228132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.228083    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386"} err="failed to get container status \"faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386\": rpc error: code = NotFound desc = could not find container \"faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386\": container with ID starting with faa69c98035dc893e2121f61047366154bf43f93a422723035edb9fbbb26f386 not found: ID does not exist"
Sep 10 17:29:10.255872 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.255845    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"]
Sep 10 17:29:10.257654 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:10.257631    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d96ccec-69df7d879d-czx26"]
Sep 10 17:29:11.114523 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:11.114490    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525c83d8-1907-425d-8cfc-a113435af06f" path="/var/lib/kubelet/pods/525c83d8-1907-425d-8cfc-a113435af06f/volumes"
Sep 10 17:29:11.114899 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:11.114875    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533c9ca5-734c-4efd-85ee-853b942031cc" path="/var/lib/kubelet/pods/533c9ca5-734c-4efd-85ee-853b942031cc/volumes"
Sep 10 17:29:18.559031 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.558998    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"]
Sep 10 17:29:18.559408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.559325    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525c83d8-1907-425d-8cfc-a113435af06f" containerName="revoke-keys"
Sep 10 17:29:18.559408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.559336    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="525c83d8-1907-425d-8cfc-a113435af06f" containerName="revoke-keys"
Sep 10 17:29:18.559408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.559350    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533c9ca5-734c-4efd-85ee-853b942031cc" containerName="payload-processing"
Sep 10 17:29:18.559408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.559358    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="533c9ca5-734c-4efd-85ee-853b942031cc" containerName="payload-processing"
Sep 10 17:29:18.559560 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.559422    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="525c83d8-1907-425d-8cfc-a113435af06f" containerName="revoke-keys"
Sep 10 17:29:18.559560 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.559434    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="533c9ca5-734c-4efd-85ee-853b942031cc" containerName="payload-processing"
Sep 10 17:29:18.562006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.561988    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.567493 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.567472    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e70241a1-dockercfg-d8hhb\""
Sep 10 17:29:18.568036 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.568007    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e70241a1\""
Sep 10 17:29:18.574832 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.574808    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"]
Sep 10 17:29:18.582744 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.582720    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"]
Sep 10 17:29:18.584927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.584913    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:18.602236 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.601799    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"]
Sep 10 17:29:18.733849 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.733816    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") " pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:18.733993 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.733857    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa0119-d905-4a89-a966-f49c917bf1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.733993 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.733931    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx48f\" (UniqueName: \"kubernetes.io/projected/12aa0119-d905-4a89-a966-f49c917bf1be-kube-api-access-vx48f\") pod \"payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.733993 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.733972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmkxw\" (UniqueName: \"kubernetes.io/projected/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-kube-api-access-mmkxw\") pod \"payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") " pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:18.835291 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.835213    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmkxw\" (UniqueName: \"kubernetes.io/projected/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-kube-api-access-mmkxw\") pod \"payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") " pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:18.835291 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.835262    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") " pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:18.835291 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.835288    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa0119-d905-4a89-a966-f49c917bf1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.835506 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.835337    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx48f\" (UniqueName: \"kubernetes.io/projected/12aa0119-d905-4a89-a966-f49c917bf1be-kube-api-access-vx48f\") pod \"payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.836053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.836031    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa0119-d905-4a89-a966-f49c917bf1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.836101 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.836057    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") " pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:18.844260 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.844234    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmkxw\" (UniqueName: \"kubernetes.io/projected/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-kube-api-access-mmkxw\") pod \"payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") " pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:18.844777 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.844761    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx48f\" (UniqueName: \"kubernetes.io/projected/12aa0119-d905-4a89-a966-f49c917bf1be-kube-api-access-vx48f\") pod \"payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.871523 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.871499    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:18.895471 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:18.895446    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:19.144752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.144726    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"]
Sep 10 17:29:19.146189 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:29:19.146167    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12aa0119_d905_4a89_a966_f49c917bf1be.slice/crio-66321580ad5b031df894e7492eae743c577dcfc6827098c69446b551950dfc6a WatchSource:0}: Error finding container 66321580ad5b031df894e7492eae743c577dcfc6827098c69446b551950dfc6a: Status 404 returned error can't find the container with id 66321580ad5b031df894e7492eae743c577dcfc6827098c69446b551950dfc6a
Sep 10 17:29:19.182341 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.182314    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"]
Sep 10 17:29:19.183212 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:29:19.183190    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77aabb92_7443_48a7_b85b_b4c2d3ffd5fb.slice/crio-ebee97d234d00c7b6cfc68b5e50229dfb698c310d27c21ac12aff7ba14710402 WatchSource:0}: Error finding container ebee97d234d00c7b6cfc68b5e50229dfb698c310d27c21ac12aff7ba14710402: Status 404 returned error can't find the container with id ebee97d234d00c7b6cfc68b5e50229dfb698c310d27c21ac12aff7ba14710402
Sep 10 17:29:19.266435 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.266395    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq" event={"ID":"12aa0119-d905-4a89-a966-f49c917bf1be","Type":"ContainerStarted","Data":"3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4"}
Sep 10 17:29:19.266620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.266446    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq" event={"ID":"12aa0119-d905-4a89-a966-f49c917bf1be","Type":"ContainerStarted","Data":"66321580ad5b031df894e7492eae743c577dcfc6827098c69446b551950dfc6a"}
Sep 10 17:29:19.266620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.266493    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:19.268240 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.268202    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6" event={"ID":"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb","Type":"ContainerStarted","Data":"15f62f2a30099545871a60ead066e2301456397a00d0a68f9ac297a81534f6d5"}
Sep 10 17:29:19.268518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.268267    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6" event={"ID":"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb","Type":"ContainerStarted","Data":"ebee97d234d00c7b6cfc68b5e50229dfb698c310d27c21ac12aff7ba14710402"}
Sep 10 17:29:19.268518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.268300    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:19.288104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.288057    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq" podStartSLOduration=1.2880402659999999 podStartE2EDuration="1.288040266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:19.286213593 +0000 UTC m=+954.642703993" watchObservedRunningTime="2026-09-10 17:29:19.288040266 +0000 UTC m=+954.644530668"
Sep 10 17:29:19.316984 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:19.316939    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6" podStartSLOduration=1.316925484 podStartE2EDuration="1.316925484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:19.315259803 +0000 UTC m=+954.671750204" watchObservedRunningTime="2026-09-10 17:29:19.316925484 +0000 UTC m=+954.673415884"
Sep 10 17:29:21.682094 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:21.682062    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:29:30.276718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:30.276682    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:30.277114 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:30.276755    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:36.982935 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:36.982899    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:29:37.360927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.360896    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"]
Sep 10 17:29:37.364258 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.364236    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.368696 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.368672    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-c38922a9-kserve-self-signed-certs\""
Sep 10 17:29:37.380431 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.380406    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"]
Sep 10 17:29:37.408835 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.408809    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-model-cache\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.408946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.408852    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-home\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.408946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.408926    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-dshm\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.408946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.408942    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7ms\" (UniqueName: \"kubernetes.io/projected/9ea2f834-65c7-4bd7-8384-40014d49ece7-kube-api-access-md7ms\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.409100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.408997    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea2f834-65c7-4bd7-8384-40014d49ece7-tls-certs\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.409100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.409081    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-tmp-dir\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.510575 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.510516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-tmp-dir\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.510752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.510681    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-model-cache\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.510752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.510741    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-home\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.510871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.510817    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-dshm\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.510871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.510845    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7ms\" (UniqueName: \"kubernetes.io/projected/9ea2f834-65c7-4bd7-8384-40014d49ece7-kube-api-access-md7ms\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.510973 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.510878    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea2f834-65c7-4bd7-8384-40014d49ece7-tls-certs\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.510973 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.510899    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-tmp-dir\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.511070 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.511051    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-model-cache\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.511129 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.511062    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-home\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.512994 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.512970    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-dshm\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.513411 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.513390    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea2f834-65c7-4bd7-8384-40014d49ece7-tls-certs\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.521882 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.521861    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7ms\" (UniqueName: \"kubernetes.io/projected/9ea2f834-65c7-4bd7-8384-40014d49ece7-kube-api-access-md7ms\") pod \"e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") " pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.673400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.673301    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:37.967212 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:37.967178    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"]
Sep 10 17:29:37.968878 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:29:37.968851    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ea2f834_65c7_4bd7_8384_40014d49ece7.slice/crio-d7fb5008b176ecd4898dd24da9d7cde6acef5ae4193a4ed8b812248e3748b3dc WatchSource:0}: Error finding container d7fb5008b176ecd4898dd24da9d7cde6acef5ae4193a4ed8b812248e3748b3dc: Status 404 returned error can't find the container with id d7fb5008b176ecd4898dd24da9d7cde6acef5ae4193a4ed8b812248e3748b3dc
Sep 10 17:29:38.363140 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:38.363102    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t" event={"ID":"9ea2f834-65c7-4bd7-8384-40014d49ece7","Type":"ContainerStarted","Data":"85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8"}
Sep 10 17:29:38.363545 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:38.363148    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t" event={"ID":"9ea2f834-65c7-4bd7-8384-40014d49ece7","Type":"ContainerStarted","Data":"d7fb5008b176ecd4898dd24da9d7cde6acef5ae4193a4ed8b812248e3748b3dc"}
Sep 10 17:29:38.387597 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:38.387549    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t" podStartSLOduration=1.387518517 podStartE2EDuration="1.387518517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:38.384462692 +0000 UTC m=+973.740953092" watchObservedRunningTime="2026-09-10 17:29:38.387518517 +0000 UTC m=+973.744008917"
Sep 10 17:29:44.807572 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:44.807539    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"]
Sep 10 17:29:44.810057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:44.810034    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"
Sep 10 17:29:44.814650 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:44.814626    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:29:44.825959 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:44.825934    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"]
Sep 10 17:29:44.885756 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:44.885725    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991465be-74cc-4d15-b019-70463eb08707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e70241a1-n5djx\" (UID: \"991465be-74cc-4d15-b019-70463eb08707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"
Sep 10 17:29:44.986619 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:44.986584    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991465be-74cc-4d15-b019-70463eb08707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e70241a1-n5djx\" (UID: \"991465be-74cc-4d15-b019-70463eb08707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"
Sep 10 17:29:44.987218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:44.987197    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991465be-74cc-4d15-b019-70463eb08707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e70241a1-n5djx\" (UID: \"991465be-74cc-4d15-b019-70463eb08707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"
Sep 10 17:29:45.120647 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:45.120572    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"
Sep 10 17:29:45.404245 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:45.404192    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"]
Sep 10 17:29:45.406601 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:29:45.406566    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991465be_74cc_4d15_b019_70463eb08707.slice/crio-c3736c20a536dad25210e14cc994617234c19925fc3574fd4871e82fafd169df WatchSource:0}: Error finding container c3736c20a536dad25210e14cc994617234c19925fc3574fd4871e82fafd169df: Status 404 returned error can't find the container with id c3736c20a536dad25210e14cc994617234c19925fc3574fd4871e82fafd169df
Sep 10 17:29:46.401485 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:46.401397    2554 generic.go:356] "Generic (PLEG): container finished" podID="991465be-74cc-4d15-b019-70463eb08707" containerID="eac25a0258617659ba9cb3023e4cd4bbe198296a06f8f07ec770f6e34d493e9a" exitCode=0
Sep 10 17:29:46.401485 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:46.401467    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx" event={"ID":"991465be-74cc-4d15-b019-70463eb08707","Type":"ContainerDied","Data":"eac25a0258617659ba9cb3023e4cd4bbe198296a06f8f07ec770f6e34d493e9a"}
Sep 10 17:29:46.401890 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:46.401503    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx" event={"ID":"991465be-74cc-4d15-b019-70463eb08707","Type":"ContainerStarted","Data":"c3736c20a536dad25210e14cc994617234c19925fc3574fd4871e82fafd169df"}
Sep 10 17:29:47.538821 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:47.538796    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"
Sep 10 17:29:47.674048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:47.673954    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:47.674048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:47.674008    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:47.688686 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:47.688661    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:47.709261 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:47.709231    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991465be-74cc-4d15-b019-70463eb08707-maas-api-service-ca\") pod \"991465be-74cc-4d15-b019-70463eb08707\" (UID: \"991465be-74cc-4d15-b019-70463eb08707\") "
Sep 10 17:29:47.709570 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:47.709540    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991465be-74cc-4d15-b019-70463eb08707-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "991465be-74cc-4d15-b019-70463eb08707" (UID: "991465be-74cc-4d15-b019-70463eb08707"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:47.709628 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:47.709617    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991465be-74cc-4d15-b019-70463eb08707-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:48.412500 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:48.412466    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx" event={"ID":"991465be-74cc-4d15-b019-70463eb08707","Type":"ContainerDied","Data":"c3736c20a536dad25210e14cc994617234c19925fc3574fd4871e82fafd169df"}
Sep 10 17:29:48.412674 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:48.412522    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3736c20a536dad25210e14cc994617234c19925fc3574fd4871e82fafd169df"
Sep 10 17:29:48.412674 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:48.412482    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"
Sep 10 17:29:48.424032 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:48.424010    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:29:51.582826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:51.582791    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:29:53.322072 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.322038    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"]
Sep 10 17:29:53.322467 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.322403    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991465be-74cc-4d15-b019-70463eb08707" containerName="revoke-keys"
Sep 10 17:29:53.322467 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.322416    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="991465be-74cc-4d15-b019-70463eb08707" containerName="revoke-keys"
Sep 10 17:29:53.322574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.322476    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="991465be-74cc-4d15-b019-70463eb08707" containerName="revoke-keys"
Sep 10 17:29:53.325311 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.325294    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.329637 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.329614    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2da043\"/\"kube-root-ca.crt\""
Sep 10 17:29:53.330467 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.330442    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2da043\"/\"autores-model-2da043-kserve-self-signed-certs\""
Sep 10 17:29:53.330467 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.330454    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2da043\"/\"openshift-service-ca.crt\""
Sep 10 17:29:53.330645 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.330472    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2da043\"/\"default-dockercfg-86tlf\""
Sep 10 17:29:53.340433 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.340404    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"]
Sep 10 17:29:53.465367 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.465331    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543391cd-d61b-49ce-b014-d4d84288973b-tls-certs\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.465367 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.465365    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-home\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.465656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.465391    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-tmp-dir\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.465656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.465462    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf8s\" (UniqueName: \"kubernetes.io/projected/543391cd-d61b-49ce-b014-d4d84288973b-kube-api-access-mpf8s\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.465656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.465548    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-dshm\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.465656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.465647    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-model-cache\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.566571 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.566510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543391cd-d61b-49ce-b014-d4d84288973b-tls-certs\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.566571 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.566562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-home\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.566571 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.566581    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-tmp-dir\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.566876 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.566599    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf8s\" (UniqueName: \"kubernetes.io/projected/543391cd-d61b-49ce-b014-d4d84288973b-kube-api-access-mpf8s\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.566876 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.566655    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-dshm\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.566876 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.566731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-model-cache\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.567150 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.567116    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-model-cache\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.567241 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.567160    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-tmp-dir\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.567333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.567309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-home\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.569207 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.569180    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543391cd-d61b-49ce-b014-d4d84288973b-tls-certs\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.569416 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.569397    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-dshm\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.576945 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.576866    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf8s\" (UniqueName: \"kubernetes.io/projected/543391cd-d61b-49ce-b014-d4d84288973b-kube-api-access-mpf8s\") pod \"autores-model-2da043-kserve-6896f99659-r94wj\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") " pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.634671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.634647    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:29:53.912403 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:53.912378    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"]
Sep 10 17:29:53.914260 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:29:53.914229    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod543391cd_d61b_49ce_b014_d4d84288973b.slice/crio-60be8ac608680eba811050570be94aae97c2f89c6f075468719aba29f24d2c89 WatchSource:0}: Error finding container 60be8ac608680eba811050570be94aae97c2f89c6f075468719aba29f24d2c89: Status 404 returned error can't find the container with id 60be8ac608680eba811050570be94aae97c2f89c6f075468719aba29f24d2c89
Sep 10 17:29:54.443022 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:54.442977    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj" event={"ID":"543391cd-d61b-49ce-b014-d4d84288973b","Type":"ContainerStarted","Data":"6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c"}
Sep 10 17:29:54.443383 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:54.443031    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj" event={"ID":"543391cd-d61b-49ce-b014-d4d84288973b","Type":"ContainerStarted","Data":"60be8ac608680eba811050570be94aae97c2f89c6f075468719aba29f24d2c89"}
Sep 10 17:29:54.466151 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:54.466104    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj" podStartSLOduration=1.466089117 podStartE2EDuration="1.466089117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:54.463095144 +0000 UTC m=+989.819585543" watchObservedRunningTime="2026-09-10 17:29:54.466089117 +0000 UTC m=+989.822579540"
Sep 10 17:29:56.333168 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.333129    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"]
Sep 10 17:29:56.333598 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.333378    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6" podUID="77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" containerName="payload-processing" containerID="cri-o://15f62f2a30099545871a60ead066e2301456397a00d0a68f9ac297a81534f6d5" gracePeriod=30
Sep 10 17:29:56.344606 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.344575    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"]
Sep 10 17:29:56.344834 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.344810    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq" podUID="12aa0119-d905-4a89-a966-f49c917bf1be" containerName="payload-pre-processing" containerID="cri-o://3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4" gracePeriod=30
Sep 10 17:29:56.454910 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.454883    2554 generic.go:356] "Generic (PLEG): container finished" podID="77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" containerID="15f62f2a30099545871a60ead066e2301456397a00d0a68f9ac297a81534f6d5" exitCode=0
Sep 10 17:29:56.455007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.454956    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6" event={"ID":"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb","Type":"ContainerDied","Data":"15f62f2a30099545871a60ead066e2301456397a00d0a68f9ac297a81534f6d5"}
Sep 10 17:29:56.546926 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.546892    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"]
Sep 10 17:29:56.554599 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.554568    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e70241a1-n5djx"]
Sep 10 17:29:56.605356 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.605326    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:56.631843 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.631816    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:56.798684 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.798655    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa0119-d905-4a89-a966-f49c917bf1be-plugins-config-volume\") pod \"12aa0119-d905-4a89-a966-f49c917bf1be\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") "
Sep 10 17:29:56.798848 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.798708    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx48f\" (UniqueName: \"kubernetes.io/projected/12aa0119-d905-4a89-a966-f49c917bf1be-kube-api-access-vx48f\") pod \"12aa0119-d905-4a89-a966-f49c917bf1be\" (UID: \"12aa0119-d905-4a89-a966-f49c917bf1be\") "
Sep 10 17:29:56.798848 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.798750    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmkxw\" (UniqueName: \"kubernetes.io/projected/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-kube-api-access-mmkxw\") pod \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") "
Sep 10 17:29:56.798994 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.798882    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-plugins-config-volume\") pod \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\" (UID: \"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb\") "
Sep 10 17:29:56.799087 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.799063    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12aa0119-d905-4a89-a966-f49c917bf1be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12aa0119-d905-4a89-a966-f49c917bf1be" (UID: "12aa0119-d905-4a89-a966-f49c917bf1be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:56.799260 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.799245    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa0119-d905-4a89-a966-f49c917bf1be-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:56.799322 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.799285    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" (UID: "77aabb92-7443-48a7-b85b-b4c2d3ffd5fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:56.800909 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.800888    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-kube-api-access-mmkxw" (OuterVolumeSpecName: "kube-api-access-mmkxw") pod "77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" (UID: "77aabb92-7443-48a7-b85b-b4c2d3ffd5fb"). InnerVolumeSpecName "kube-api-access-mmkxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:56.800988 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.800932    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12aa0119-d905-4a89-a966-f49c917bf1be-kube-api-access-vx48f" (OuterVolumeSpecName: "kube-api-access-vx48f") pod "12aa0119-d905-4a89-a966-f49c917bf1be" (UID: "12aa0119-d905-4a89-a966-f49c917bf1be"). InnerVolumeSpecName "kube-api-access-vx48f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:56.900671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.900593    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:56.900671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.900620    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx48f\" (UniqueName: \"kubernetes.io/projected/12aa0119-d905-4a89-a966-f49c917bf1be-kube-api-access-vx48f\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:56.900671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:56.900634    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmkxw\" (UniqueName: \"kubernetes.io/projected/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb-kube-api-access-mmkxw\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:29:57.114000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.113955    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991465be-74cc-4d15-b019-70463eb08707" path="/var/lib/kubelet/pods/991465be-74cc-4d15-b019-70463eb08707/volumes"
Sep 10 17:29:57.464184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.464146    2554 generic.go:356] "Generic (PLEG): container finished" podID="12aa0119-d905-4a89-a966-f49c917bf1be" containerID="3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4" exitCode=0
Sep 10 17:29:57.464676 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.464211    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"
Sep 10 17:29:57.464676 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.464231    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq" event={"ID":"12aa0119-d905-4a89-a966-f49c917bf1be","Type":"ContainerDied","Data":"3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4"}
Sep 10 17:29:57.464676 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.464335    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq" event={"ID":"12aa0119-d905-4a89-a966-f49c917bf1be","Type":"ContainerDied","Data":"66321580ad5b031df894e7492eae743c577dcfc6827098c69446b551950dfc6a"}
Sep 10 17:29:57.464676 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.464366    2554 scope.go:117] "RemoveContainer" containerID="3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4"
Sep 10 17:29:57.465811 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.465787    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6" event={"ID":"77aabb92-7443-48a7-b85b-b4c2d3ffd5fb","Type":"ContainerDied","Data":"ebee97d234d00c7b6cfc68b5e50229dfb698c310d27c21ac12aff7ba14710402"}
Sep 10 17:29:57.465891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.465845    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"
Sep 10 17:29:57.473192 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.473178    2554 scope.go:117] "RemoveContainer" containerID="3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4"
Sep 10 17:29:57.473472 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:29:57.473452    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4\": container with ID starting with 3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4 not found: ID does not exist" containerID="3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4"
Sep 10 17:29:57.473651 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.473481    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4"} err="failed to get container status \"3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4\": rpc error: code = NotFound desc = could not find container \"3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4\": container with ID starting with 3e56f9c233fcf1baa626a78c4b5522496ff413789195e2a3518d61cf19a4c1d4 not found: ID does not exist"
Sep 10 17:29:57.473651 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.473497    2554 scope.go:117] "RemoveContainer" containerID="15f62f2a30099545871a60ead066e2301456397a00d0a68f9ac297a81534f6d5"
Sep 10 17:29:57.488131 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.488108    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"]
Sep 10 17:29:57.494249 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.494229    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e70241a1-7b76cc7fc4-52nvq"]
Sep 10 17:29:57.512581 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.512552    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"]
Sep 10 17:29:57.517804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:57.517782    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e70241a1-6db77479df-2hqg6"]
Sep 10 17:29:59.114692 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:59.114654    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12aa0119-d905-4a89-a966-f49c917bf1be" path="/var/lib/kubelet/pods/12aa0119-d905-4a89-a966-f49c917bf1be/volumes"
Sep 10 17:29:59.115054 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:29:59.115032    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" path="/var/lib/kubelet/pods/77aabb92-7443-48a7-b85b-b4c2d3ffd5fb/volumes"
Sep 10 17:30:00.167738 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.167702    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5"]
Sep 10 17:30:00.168227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.168205    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" containerName="payload-processing"
Sep 10 17:30:00.168305 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.168231    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" containerName="payload-processing"
Sep 10 17:30:00.168305 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.168279    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12aa0119-d905-4a89-a966-f49c917bf1be" containerName="payload-pre-processing"
Sep 10 17:30:00.168305 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.168288    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="12aa0119-d905-4a89-a966-f49c917bf1be" containerName="payload-pre-processing"
Sep 10 17:30:00.168454 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.168377    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="12aa0119-d905-4a89-a966-f49c917bf1be" containerName="payload-pre-processing"
Sep 10 17:30:00.168454 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.168392    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="77aabb92-7443-48a7-b85b-b4c2d3ffd5fb" containerName="payload-processing"
Sep 10 17:30:00.172952 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.172930    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5"
Sep 10 17:30:00.175910 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.175887    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"]
Sep 10 17:30:00.178248 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.178227    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:30:00.179151 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.179136    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"
Sep 10 17:30:00.182389 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.182366    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5"
Sep 10 17:30:00.189886 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.189871    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"
Sep 10 17:30:00.197795 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.197771    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5"]
Sep 10 17:30:00.204654 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.204628    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"]
Sep 10 17:30:00.471215 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.471185    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5"]
Sep 10 17:30:00.473037 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:30:00.473010    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7877d0a2_acf9_476a_a048_b54896f5f870.slice/crio-4813ae2b5c5bb2d1f93ce5339ccc575df2b15a97b192fce2015a566b7935e93d WatchSource:0}: Error finding container 4813ae2b5c5bb2d1f93ce5339ccc575df2b15a97b192fce2015a566b7935e93d: Status 404 returned error can't find the container with id 4813ae2b5c5bb2d1f93ce5339ccc575df2b15a97b192fce2015a566b7935e93d
Sep 10 17:30:00.484322 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.484299    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5" event={"ID":"7877d0a2-acf9-476a-a048-b54896f5f870","Type":"ContainerStarted","Data":"4813ae2b5c5bb2d1f93ce5339ccc575df2b15a97b192fce2015a566b7935e93d"}
Sep 10 17:30:00.488432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:00.488333    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"]
Sep 10 17:30:00.490239 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:30:00.490210    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d97b3a0_a60a_45d7_85a2_1debe9a16a65.slice/crio-a965411387be68e709634bb021936a2ba8501aab9cd4a2f068cd4baf8e240ef5 WatchSource:0}: Error finding container a965411387be68e709634bb021936a2ba8501aab9cd4a2f068cd4baf8e240ef5: Status 404 returned error can't find the container with id a965411387be68e709634bb021936a2ba8501aab9cd4a2f068cd4baf8e240ef5
Sep 10 17:30:01.490936 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:01.490902    2554 generic.go:356] "Generic (PLEG): container finished" podID="7877d0a2-acf9-476a-a048-b54896f5f870" containerID="5f29174abab8de9bbbe00afc835a18bb2d3794a4093f319fe66a7e6dc6254efe" exitCode=0
Sep 10 17:30:01.491333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:01.490954    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5" event={"ID":"7877d0a2-acf9-476a-a048-b54896f5f870","Type":"ContainerDied","Data":"5f29174abab8de9bbbe00afc835a18bb2d3794a4093f319fe66a7e6dc6254efe"}
Sep 10 17:30:01.492508 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:01.492485    2554 generic.go:356] "Generic (PLEG): container finished" podID="1d97b3a0-a60a-45d7-85a2-1debe9a16a65" containerID="85ee014cbc019c4bc119f4615b924a2d2cfd6df1fee4584a94c7c8b9faf982e5" exitCode=0
Sep 10 17:30:01.492628 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:01.492556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt" event={"ID":"1d97b3a0-a60a-45d7-85a2-1debe9a16a65","Type":"ContainerDied","Data":"85ee014cbc019c4bc119f4615b924a2d2cfd6df1fee4584a94c7c8b9faf982e5"}
Sep 10 17:30:01.492628 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:01.492597    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt" event={"ID":"1d97b3a0-a60a-45d7-85a2-1debe9a16a65","Type":"ContainerStarted","Data":"a965411387be68e709634bb021936a2ba8501aab9cd4a2f068cd4baf8e240ef5"}
Sep 10 17:30:02.641106 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:02.641079    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5"
Sep 10 17:30:02.669275 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:02.669228    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"
Sep 10 17:30:03.505962 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.505864    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5"
Sep 10 17:30:03.505962 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.505881    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817690-8sdc5" event={"ID":"7877d0a2-acf9-476a-a048-b54896f5f870","Type":"ContainerDied","Data":"4813ae2b5c5bb2d1f93ce5339ccc575df2b15a97b192fce2015a566b7935e93d"}
Sep 10 17:30:03.505962 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.505929    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4813ae2b5c5bb2d1f93ce5339ccc575df2b15a97b192fce2015a566b7935e93d"
Sep 10 17:30:03.507297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.507278    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"
Sep 10 17:30:03.507513 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.507280    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt" event={"ID":"1d97b3a0-a60a-45d7-85a2-1debe9a16a65","Type":"ContainerDied","Data":"a965411387be68e709634bb021936a2ba8501aab9cd4a2f068cd4baf8e240ef5"}
Sep 10 17:30:03.507513 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.507393    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a965411387be68e709634bb021936a2ba8501aab9cd4a2f068cd4baf8e240ef5"
Sep 10 17:30:03.634985 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.634953    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:30:03.635150 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.635018    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:30:03.647415 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:03.647387    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:30:04.523517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:04.523487    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:30:13.397589 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:13.397549    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:30:17.727384 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.727347    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"]
Sep 10 17:30:17.727790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.727739    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7877d0a2-acf9-476a-a048-b54896f5f870" containerName="cleanup"
Sep 10 17:30:17.727790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.727752    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7877d0a2-acf9-476a-a048-b54896f5f870" containerName="cleanup"
Sep 10 17:30:17.727790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.727761    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d97b3a0-a60a-45d7-85a2-1debe9a16a65" containerName="cleanup"
Sep 10 17:30:17.727790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.727768    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d97b3a0-a60a-45d7-85a2-1debe9a16a65" containerName="cleanup"
Sep 10 17:30:17.727940 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.727823    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d97b3a0-a60a-45d7-85a2-1debe9a16a65" containerName="cleanup"
Sep 10 17:30:17.727940 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.727833    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7877d0a2-acf9-476a-a048-b54896f5f870" containerName="cleanup"
Sep 10 17:30:17.731152 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.731136    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.735468 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.735447    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8c46e8\"/\"openshift-service-ca.crt\""
Sep 10 17:30:17.736685 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.736657    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8c46e8\"/\"kube-root-ca.crt\""
Sep 10 17:30:17.736803 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.736732    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8c46e8\"/\"default-dockercfg-c5h9b\""
Sep 10 17:30:17.736863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.736845    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8c46e8\"/\"test-model-8c46e8-kserve-self-signed-certs\""
Sep 10 17:30:17.748004 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.747976    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"]
Sep 10 17:30:17.806250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.806215    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-home\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.806250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.806249    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-dshm\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.806432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.806276    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvr9\" (UniqueName: \"kubernetes.io/projected/14955e52-c8a8-40b7-91e0-55421e9fa3a6-kube-api-access-8zvr9\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.806432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.806364    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tmp-dir\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.806432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.806385    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-model-cache\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.806432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.806409    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tls-certs\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907190 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907156    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tls-certs\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907354 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907225    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-home\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907354 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907247    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-dshm\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907354 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907275    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvr9\" (UniqueName: \"kubernetes.io/projected/14955e52-c8a8-40b7-91e0-55421e9fa3a6-kube-api-access-8zvr9\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907354 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tmp-dir\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907582 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907360    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-model-cache\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907839 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907803    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tmp-dir\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.907839 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907831    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-home\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.908024 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.907850    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-model-cache\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.909572 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.909544    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-dshm\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.909751 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.909735    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tls-certs\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:17.922110 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:17.922085    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvr9\" (UniqueName: \"kubernetes.io/projected/14955e52-c8a8-40b7-91e0-55421e9fa3a6-kube-api-access-8zvr9\") pod \"test-model-8c46e8-kserve-b5ccbf9f7-99fls\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") " pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:18.040408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:18.040316    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:18.316592 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:18.316565    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"]
Sep 10 17:30:18.317989 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:30:18.317963    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14955e52_c8a8_40b7_91e0_55421e9fa3a6.slice/crio-c9eaf16e50087d4ca4d8704876cdc14ed289f85ffe6969819b3ded032eadfa80 WatchSource:0}: Error finding container c9eaf16e50087d4ca4d8704876cdc14ed289f85ffe6969819b3ded032eadfa80: Status 404 returned error can't find the container with id c9eaf16e50087d4ca4d8704876cdc14ed289f85ffe6969819b3ded032eadfa80
Sep 10 17:30:18.581166 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:18.581083    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls" event={"ID":"14955e52-c8a8-40b7-91e0-55421e9fa3a6","Type":"ContainerStarted","Data":"958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f"}
Sep 10 17:30:18.581166 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:18.581127    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls" event={"ID":"14955e52-c8a8-40b7-91e0-55421e9fa3a6","Type":"ContainerStarted","Data":"c9eaf16e50087d4ca4d8704876cdc14ed289f85ffe6969819b3ded032eadfa80"}
Sep 10 17:30:18.611785 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:18.611739    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls" podStartSLOduration=1.6117243650000002 podStartE2EDuration="1.611724365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:18.610973831 +0000 UTC m=+1013.967464234" watchObservedRunningTime="2026-09-10 17:30:18.611724365 +0000 UTC m=+1013.968214764"
Sep 10 17:30:19.897297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:19.897266    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"]
Sep 10 17:30:20.212104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:20.212031    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"]
Sep 10 17:30:20.959263 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:20.959217    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"]
Sep 10 17:30:21.123552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.123508    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"]
Sep 10 17:30:21.126965 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.126949    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"
Sep 10 17:30:21.129788 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.129766    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:30:21.138294 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.138266    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"]
Sep 10 17:30:21.239275 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.239186    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba4c2a6-cae0-4f76-b958-1236b73a2846-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2da043-b6ntb\" (UID: \"9ba4c2a6-cae0-4f76-b958-1236b73a2846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"
Sep 10 17:30:21.340416 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.340376    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba4c2a6-cae0-4f76-b958-1236b73a2846-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2da043-b6ntb\" (UID: \"9ba4c2a6-cae0-4f76-b958-1236b73a2846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"
Sep 10 17:30:21.341057 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.341038    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba4c2a6-cae0-4f76-b958-1236b73a2846-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2da043-b6ntb\" (UID: \"9ba4c2a6-cae0-4f76-b958-1236b73a2846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"
Sep 10 17:30:21.438651 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.438619    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"
Sep 10 17:30:21.712877 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:21.712853    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"]
Sep 10 17:30:21.714900 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:30:21.714875    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba4c2a6_cae0_4f76_b958_1236b73a2846.slice/crio-09def1ddec8851d6df776ccff7d04b420502ed33d361791c009eb230a83f6472 WatchSource:0}: Error finding container 09def1ddec8851d6df776ccff7d04b420502ed33d361791c009eb230a83f6472: Status 404 returned error can't find the container with id 09def1ddec8851d6df776ccff7d04b420502ed33d361791c009eb230a83f6472
Sep 10 17:30:22.602065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:22.602027    2554 generic.go:356] "Generic (PLEG): container finished" podID="9ba4c2a6-cae0-4f76-b958-1236b73a2846" containerID="3dd7d5e23cd34caa6ad542178afdd59acab04dd4ca952c43b6a6460255c16fad" exitCode=0
Sep 10 17:30:22.602421 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:22.602112    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb" event={"ID":"9ba4c2a6-cae0-4f76-b958-1236b73a2846","Type":"ContainerDied","Data":"3dd7d5e23cd34caa6ad542178afdd59acab04dd4ca952c43b6a6460255c16fad"}
Sep 10 17:30:22.602421 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:22.602167    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb" event={"ID":"9ba4c2a6-cae0-4f76-b958-1236b73a2846","Type":"ContainerStarted","Data":"09def1ddec8851d6df776ccff7d04b420502ed33d361791c009eb230a83f6472"}
Sep 10 17:30:23.748870 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:23.748845    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"
Sep 10 17:30:23.864259 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:23.864173    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba4c2a6-cae0-4f76-b958-1236b73a2846-maas-api-service-ca\") pod \"9ba4c2a6-cae0-4f76-b958-1236b73a2846\" (UID: \"9ba4c2a6-cae0-4f76-b958-1236b73a2846\") "
Sep 10 17:30:23.864588 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:23.864552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba4c2a6-cae0-4f76-b958-1236b73a2846-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ba4c2a6-cae0-4f76-b958-1236b73a2846" (UID: "9ba4c2a6-cae0-4f76-b958-1236b73a2846"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:30:23.965824 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:23.965789    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba4c2a6-cae0-4f76-b958-1236b73a2846-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:24.614192 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:24.614164    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"
Sep 10 17:30:24.614375 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:24.614205    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb" event={"ID":"9ba4c2a6-cae0-4f76-b958-1236b73a2846","Type":"ContainerDied","Data":"09def1ddec8851d6df776ccff7d04b420502ed33d361791c009eb230a83f6472"}
Sep 10 17:30:24.614375 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:24.614259    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09def1ddec8851d6df776ccff7d04b420502ed33d361791c009eb230a83f6472"
Sep 10 17:30:28.040920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:28.040885    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:28.041385 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:28.040940    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:28.053582 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:28.053557    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:28.645731 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:28.645695    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:30:30.672037 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.671997    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"]
Sep 10 17:30:30.672555 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.672356    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba4c2a6-cae0-4f76-b958-1236b73a2846" containerName="revoke-keys"
Sep 10 17:30:30.672555 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.672368    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba4c2a6-cae0-4f76-b958-1236b73a2846" containerName="revoke-keys"
Sep 10 17:30:30.672555 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.672425    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba4c2a6-cae0-4f76-b958-1236b73a2846" containerName="revoke-keys"
Sep 10 17:30:30.676688 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.676667    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"
Sep 10 17:30:30.679563 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.679546    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:30:30.688964 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.688941    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"]
Sep 10 17:30:30.730642 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.730613    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95987c-3c8c-4c2f-b3ab-ead49ca37398-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc\" (UID: \"ff95987c-3c8c-4c2f-b3ab-ead49ca37398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"
Sep 10 17:30:30.831227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.831198    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95987c-3c8c-4c2f-b3ab-ead49ca37398-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc\" (UID: \"ff95987c-3c8c-4c2f-b3ab-ead49ca37398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"
Sep 10 17:30:30.831848 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.831830    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95987c-3c8c-4c2f-b3ab-ead49ca37398-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc\" (UID: \"ff95987c-3c8c-4c2f-b3ab-ead49ca37398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"
Sep 10 17:30:30.986574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:30.986474    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"
Sep 10 17:30:31.262465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:31.262438    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"]
Sep 10 17:30:31.264129 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:30:31.264098    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff95987c_3c8c_4c2f_b3ab_ead49ca37398.slice/crio-3d6b6cc6423ce4c1a1775a30c7d2bcb9f323850d0dc3a2968205beefa6e45420 WatchSource:0}: Error finding container 3d6b6cc6423ce4c1a1775a30c7d2bcb9f323850d0dc3a2968205beefa6e45420: Status 404 returned error can't find the container with id 3d6b6cc6423ce4c1a1775a30c7d2bcb9f323850d0dc3a2968205beefa6e45420
Sep 10 17:30:31.652180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:31.652144    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc" event={"ID":"ff95987c-3c8c-4c2f-b3ab-ead49ca37398","Type":"ContainerStarted","Data":"27b91b486b378f16d11968a3c39f0b859cc88e46930f839eb992365d0b564907"}
Sep 10 17:30:31.652343 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:31.652205    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc" event={"ID":"ff95987c-3c8c-4c2f-b3ab-ead49ca37398","Type":"ContainerStarted","Data":"3d6b6cc6423ce4c1a1775a30c7d2bcb9f323850d0dc3a2968205beefa6e45420"}
Sep 10 17:30:31.681713 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:31.681667    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc" podStartSLOduration=1.6816516460000002 podStartE2EDuration="1.681651646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:31.680890578 +0000 UTC m=+1027.037380980" watchObservedRunningTime="2026-09-10 17:30:31.681651646 +0000 UTC m=+1027.038142052"
Sep 10 17:30:32.659464 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:32.659426    2554 generic.go:356] "Generic (PLEG): container finished" podID="ff95987c-3c8c-4c2f-b3ab-ead49ca37398" containerID="27b91b486b378f16d11968a3c39f0b859cc88e46930f839eb992365d0b564907" exitCode=0
Sep 10 17:30:32.659659 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:32.659512    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc" event={"ID":"ff95987c-3c8c-4c2f-b3ab-ead49ca37398","Type":"ContainerDied","Data":"27b91b486b378f16d11968a3c39f0b859cc88e46930f839eb992365d0b564907"}
Sep 10 17:30:33.797062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:33.797027    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"
Sep 10 17:30:33.859625 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:33.859586    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95987c-3c8c-4c2f-b3ab-ead49ca37398-maas-api-service-ca\") pod \"ff95987c-3c8c-4c2f-b3ab-ead49ca37398\" (UID: \"ff95987c-3c8c-4c2f-b3ab-ead49ca37398\") "
Sep 10 17:30:33.860005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:33.859978    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff95987c-3c8c-4c2f-b3ab-ead49ca37398-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff95987c-3c8c-4c2f-b3ab-ead49ca37398" (UID: "ff95987c-3c8c-4c2f-b3ab-ead49ca37398"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:30:33.960979 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:33.960894    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95987c-3c8c-4c2f-b3ab-ead49ca37398-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:34.476537 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:34.476502    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"]
Sep 10 17:30:34.483937 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:34.483908    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2da043-b6ntb"]
Sep 10 17:30:34.599240 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:34.599203    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:30:34.676420 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:34.676375    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc" event={"ID":"ff95987c-3c8c-4c2f-b3ab-ead49ca37398","Type":"ContainerDied","Data":"3d6b6cc6423ce4c1a1775a30c7d2bcb9f323850d0dc3a2968205beefa6e45420"}
Sep 10 17:30:34.676587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:34.676447    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d6b6cc6423ce4c1a1775a30c7d2bcb9f323850d0dc3a2968205beefa6e45420"
Sep 10 17:30:34.676587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:34.676398    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"
Sep 10 17:30:34.916581 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:34.916519    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks" podUID="fba63699-d7b5-4fff-ade7-1afcc4d62684" containerName="main" containerID="cri-o://00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab" gracePeriod=45
Sep 10 17:30:35.118117 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.118079    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba4c2a6-cae0-4f76-b958-1236b73a2846" path="/var/lib/kubelet/pods/9ba4c2a6-cae0-4f76-b958-1236b73a2846/volumes"
Sep 10 17:30:35.164955 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.164930    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:30:35.231248 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.231164    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t" podUID="9ea2f834-65c7-4bd7-8384-40014d49ece7" containerName="main" containerID="cri-o://85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8" gracePeriod=45
Sep 10 17:30:35.273355 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273328    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-model-cache\") pod \"fba63699-d7b5-4fff-ade7-1afcc4d62684\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") "
Sep 10 17:30:35.273486 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273381    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba63699-d7b5-4fff-ade7-1afcc4d62684-tls-certs\") pod \"fba63699-d7b5-4fff-ade7-1afcc4d62684\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") "
Sep 10 17:30:35.273486 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273459    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-home\") pod \"fba63699-d7b5-4fff-ade7-1afcc4d62684\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") "
Sep 10 17:30:35.273622 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273498    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtkmw\" (UniqueName: \"kubernetes.io/projected/fba63699-d7b5-4fff-ade7-1afcc4d62684-kube-api-access-mtkmw\") pod \"fba63699-d7b5-4fff-ade7-1afcc4d62684\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") "
Sep 10 17:30:35.273622 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273598    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-tmp-dir\") pod \"fba63699-d7b5-4fff-ade7-1afcc4d62684\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") "
Sep 10 17:30:35.273726 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273616    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-model-cache" (OuterVolumeSpecName: "model-cache") pod "fba63699-d7b5-4fff-ade7-1afcc4d62684" (UID: "fba63699-d7b5-4fff-ade7-1afcc4d62684"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.273726 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273693    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-dshm\") pod \"fba63699-d7b5-4fff-ade7-1afcc4d62684\" (UID: \"fba63699-d7b5-4fff-ade7-1afcc4d62684\") "
Sep 10 17:30:35.273844 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273802    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-home" (OuterVolumeSpecName: "home") pod "fba63699-d7b5-4fff-ade7-1afcc4d62684" (UID: "fba63699-d7b5-4fff-ade7-1afcc4d62684"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.273899 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.273875    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fba63699-d7b5-4fff-ade7-1afcc4d62684" (UID: "fba63699-d7b5-4fff-ade7-1afcc4d62684"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.274091 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.274054    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.274091 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.274078    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.274221 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.274091    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.275883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.275857    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-dshm" (OuterVolumeSpecName: "dshm") pod "fba63699-d7b5-4fff-ade7-1afcc4d62684" (UID: "fba63699-d7b5-4fff-ade7-1afcc4d62684"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.275990 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.275939    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba63699-d7b5-4fff-ade7-1afcc4d62684-kube-api-access-mtkmw" (OuterVolumeSpecName: "kube-api-access-mtkmw") pod "fba63699-d7b5-4fff-ade7-1afcc4d62684" (UID: "fba63699-d7b5-4fff-ade7-1afcc4d62684"). InnerVolumeSpecName "kube-api-access-mtkmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:30:35.276043 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.275982    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba63699-d7b5-4fff-ade7-1afcc4d62684-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fba63699-d7b5-4fff-ade7-1afcc4d62684" (UID: "fba63699-d7b5-4fff-ade7-1afcc4d62684"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:30:35.374888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.374859    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba63699-d7b5-4fff-ade7-1afcc4d62684-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.374888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.374887    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba63699-d7b5-4fff-ade7-1afcc4d62684-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.375050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.374897    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkmw\" (UniqueName: \"kubernetes.io/projected/fba63699-d7b5-4fff-ade7-1afcc4d62684-kube-api-access-mtkmw\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.466725 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.466698    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:30:35.576153 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576126    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-dshm\") pod \"9ea2f834-65c7-4bd7-8384-40014d49ece7\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") "
Sep 10 17:30:35.576294 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576157    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-tmp-dir\") pod \"9ea2f834-65c7-4bd7-8384-40014d49ece7\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") "
Sep 10 17:30:35.576294 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576183    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md7ms\" (UniqueName: \"kubernetes.io/projected/9ea2f834-65c7-4bd7-8384-40014d49ece7-kube-api-access-md7ms\") pod \"9ea2f834-65c7-4bd7-8384-40014d49ece7\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") "
Sep 10 17:30:35.576294 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576214    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea2f834-65c7-4bd7-8384-40014d49ece7-tls-certs\") pod \"9ea2f834-65c7-4bd7-8384-40014d49ece7\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") "
Sep 10 17:30:35.576294 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576260    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-model-cache\") pod \"9ea2f834-65c7-4bd7-8384-40014d49ece7\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") "
Sep 10 17:30:35.576543 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576314    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-home\") pod \"9ea2f834-65c7-4bd7-8384-40014d49ece7\" (UID: \"9ea2f834-65c7-4bd7-8384-40014d49ece7\") "
Sep 10 17:30:35.576543 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576430    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ea2f834-65c7-4bd7-8384-40014d49ece7" (UID: "9ea2f834-65c7-4bd7-8384-40014d49ece7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.576664 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576517    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ea2f834-65c7-4bd7-8384-40014d49ece7" (UID: "9ea2f834-65c7-4bd7-8384-40014d49ece7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.576664 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576612    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-home" (OuterVolumeSpecName: "home") pod "9ea2f834-65c7-4bd7-8384-40014d49ece7" (UID: "9ea2f834-65c7-4bd7-8384-40014d49ece7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.576781 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576762    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.576830 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576788    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.576830 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.576800    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.578361 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.578338    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea2f834-65c7-4bd7-8384-40014d49ece7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ea2f834-65c7-4bd7-8384-40014d49ece7" (UID: "9ea2f834-65c7-4bd7-8384-40014d49ece7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:30:35.578732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.578716    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-dshm" (OuterVolumeSpecName: "dshm") pod "9ea2f834-65c7-4bd7-8384-40014d49ece7" (UID: "9ea2f834-65c7-4bd7-8384-40014d49ece7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:35.578787 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.578733    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea2f834-65c7-4bd7-8384-40014d49ece7-kube-api-access-md7ms" (OuterVolumeSpecName: "kube-api-access-md7ms") pod "9ea2f834-65c7-4bd7-8384-40014d49ece7" (UID: "9ea2f834-65c7-4bd7-8384-40014d49ece7"). InnerVolumeSpecName "kube-api-access-md7ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:30:35.677697 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.677670    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ea2f834-65c7-4bd7-8384-40014d49ece7-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.677697 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.677693    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md7ms\" (UniqueName: \"kubernetes.io/projected/9ea2f834-65c7-4bd7-8384-40014d49ece7-kube-api-access-md7ms\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.677854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.677702    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ea2f834-65c7-4bd7-8384-40014d49ece7-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:35.682986 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.682962    2554 generic.go:356] "Generic (PLEG): container finished" podID="fba63699-d7b5-4fff-ade7-1afcc4d62684" containerID="00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab" exitCode=0
Sep 10 17:30:35.683092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.683009    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"
Sep 10 17:30:35.683092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.683044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks" event={"ID":"fba63699-d7b5-4fff-ade7-1afcc4d62684","Type":"ContainerDied","Data":"00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab"}
Sep 10 17:30:35.683190 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.683090    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks" event={"ID":"fba63699-d7b5-4fff-ade7-1afcc4d62684","Type":"ContainerDied","Data":"3d501933092e04710ecdca3abbaa3420a0b6e3c09151edd9b4b66da61130c8db"}
Sep 10 17:30:35.683190 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.683118    2554 scope.go:117] "RemoveContainer" containerID="00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab"
Sep 10 17:30:35.684504 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.684484    2554 generic.go:356] "Generic (PLEG): container finished" podID="9ea2f834-65c7-4bd7-8384-40014d49ece7" containerID="85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8" exitCode=0
Sep 10 17:30:35.684613 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.684523    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"
Sep 10 17:30:35.684613 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.684557    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t" event={"ID":"9ea2f834-65c7-4bd7-8384-40014d49ece7","Type":"ContainerDied","Data":"85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8"}
Sep 10 17:30:35.684613 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.684598    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t" event={"ID":"9ea2f834-65c7-4bd7-8384-40014d49ece7","Type":"ContainerDied","Data":"d7fb5008b176ecd4898dd24da9d7cde6acef5ae4193a4ed8b812248e3748b3dc"}
Sep 10 17:30:35.692164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.692065    2554 scope.go:117] "RemoveContainer" containerID="00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab"
Sep 10 17:30:35.692446 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:30:35.692418    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab\": container with ID starting with 00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab not found: ID does not exist" containerID="00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab"
Sep 10 17:30:35.692507 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.692457    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab"} err="failed to get container status \"00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab\": rpc error: code = NotFound desc = could not find container \"00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab\": container with ID starting with 00d1fbcc4a4498b6e28768d7eec489256860eff8d3a7aa07c4cd9fbdef0e8fab not found: ID does not exist"
Sep 10 17:30:35.692507 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.692480    2554 scope.go:117] "RemoveContainer" containerID="85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8"
Sep 10 17:30:35.700185 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.700169    2554 scope.go:117] "RemoveContainer" containerID="85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8"
Sep 10 17:30:35.700427 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:30:35.700413    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8\": container with ID starting with 85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8 not found: ID does not exist" containerID="85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8"
Sep 10 17:30:35.700470 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.700434    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8"} err="failed to get container status \"85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8\": rpc error: code = NotFound desc = could not find container \"85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8\": container with ID starting with 85ee1a7be1d4af8b74ccba48b2e046d4bf637499d8db76ffe5e2dbfe19fa16b8 not found: ID does not exist"
Sep 10 17:30:35.715157 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.715134    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"]
Sep 10 17:30:35.725896 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.725876    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-c38922a9-kserve-649ccf476b-qbw5t"]
Sep 10 17:30:35.750540 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.750507    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"]
Sep 10 17:30:35.756936 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.756906    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-c38922a9-kserve-67f4b7575b-c69ks"]
Sep 10 17:30:35.979063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:35.979022    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj" podUID="543391cd-d61b-49ce-b014-d4d84288973b" containerName="main" containerID="cri-o://6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c" gracePeriod=45
Sep 10 17:30:36.215965 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.215938    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:30:36.282928 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.282899    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-dshm\") pod \"543391cd-d61b-49ce-b014-d4d84288973b\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") "
Sep 10 17:30:36.283071 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.282956    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpf8s\" (UniqueName: \"kubernetes.io/projected/543391cd-d61b-49ce-b014-d4d84288973b-kube-api-access-mpf8s\") pod \"543391cd-d61b-49ce-b014-d4d84288973b\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") "
Sep 10 17:30:36.283132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283073    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-home\") pod \"543391cd-d61b-49ce-b014-d4d84288973b\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") "
Sep 10 17:30:36.283132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283113    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-tmp-dir\") pod \"543391cd-d61b-49ce-b014-d4d84288973b\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") "
Sep 10 17:30:36.283241 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283163    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-model-cache\") pod \"543391cd-d61b-49ce-b014-d4d84288973b\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") "
Sep 10 17:30:36.283241 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283189    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543391cd-d61b-49ce-b014-d4d84288973b-tls-certs\") pod \"543391cd-d61b-49ce-b014-d4d84288973b\" (UID: \"543391cd-d61b-49ce-b014-d4d84288973b\") "
Sep 10 17:30:36.283439 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283361    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-home" (OuterVolumeSpecName: "home") pod "543391cd-d61b-49ce-b014-d4d84288973b" (UID: "543391cd-d61b-49ce-b014-d4d84288973b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:36.283439 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283417    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "543391cd-d61b-49ce-b014-d4d84288973b" (UID: "543391cd-d61b-49ce-b014-d4d84288973b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:36.283632 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283434    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-model-cache" (OuterVolumeSpecName: "model-cache") pod "543391cd-d61b-49ce-b014-d4d84288973b" (UID: "543391cd-d61b-49ce-b014-d4d84288973b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:36.283632 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283574    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:36.283632 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283596    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:36.283632 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.283609    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:36.285306 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.285278    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-dshm" (OuterVolumeSpecName: "dshm") pod "543391cd-d61b-49ce-b014-d4d84288973b" (UID: "543391cd-d61b-49ce-b014-d4d84288973b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:30:36.285398 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.285327    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543391cd-d61b-49ce-b014-d4d84288973b-kube-api-access-mpf8s" (OuterVolumeSpecName: "kube-api-access-mpf8s") pod "543391cd-d61b-49ce-b014-d4d84288973b" (UID: "543391cd-d61b-49ce-b014-d4d84288973b"). InnerVolumeSpecName "kube-api-access-mpf8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:30:36.285398 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.285333    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543391cd-d61b-49ce-b014-d4d84288973b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "543391cd-d61b-49ce-b014-d4d84288973b" (UID: "543391cd-d61b-49ce-b014-d4d84288973b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:30:36.385062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.385030    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543391cd-d61b-49ce-b014-d4d84288973b-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:36.385062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.385053    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543391cd-d61b-49ce-b014-d4d84288973b-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:36.385062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.385062    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpf8s\" (UniqueName: \"kubernetes.io/projected/543391cd-d61b-49ce-b014-d4d84288973b-kube-api-access-mpf8s\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:30:36.692794 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.692753    2554 generic.go:356] "Generic (PLEG): container finished" podID="543391cd-d61b-49ce-b014-d4d84288973b" containerID="6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c" exitCode=0
Sep 10 17:30:36.692943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.692801    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"
Sep 10 17:30:36.692943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.692843    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj" event={"ID":"543391cd-d61b-49ce-b014-d4d84288973b","Type":"ContainerDied","Data":"6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c"}
Sep 10 17:30:36.692943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.692884    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj" event={"ID":"543391cd-d61b-49ce-b014-d4d84288973b","Type":"ContainerDied","Data":"60be8ac608680eba811050570be94aae97c2f89c6f075468719aba29f24d2c89"}
Sep 10 17:30:36.692943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.692902    2554 scope.go:117] "RemoveContainer" containerID="6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c"
Sep 10 17:30:36.702300 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.702285    2554 scope.go:117] "RemoveContainer" containerID="6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c"
Sep 10 17:30:36.702584 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:30:36.702566    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c\": container with ID starting with 6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c not found: ID does not exist" containerID="6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c"
Sep 10 17:30:36.702631 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.702592    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c"} err="failed to get container status \"6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c\": rpc error: code = NotFound desc = could not find container \"6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c\": container with ID starting with 6d59d3c5e3ef1441252074350281cf3530fbc0e50eace6b03d5c69e29433c78c not found: ID does not exist"
Sep 10 17:30:36.722812 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.722785    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"]
Sep 10 17:30:36.731320 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:36.731298    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2da043/autores-model-2da043-kserve-6896f99659-r94wj"]
Sep 10 17:30:37.114699 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:37.114664    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543391cd-d61b-49ce-b014-d4d84288973b" path="/var/lib/kubelet/pods/543391cd-d61b-49ce-b014-d4d84288973b/volumes"
Sep 10 17:30:37.115084 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:37.115071    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea2f834-65c7-4bd7-8384-40014d49ece7" path="/var/lib/kubelet/pods/9ea2f834-65c7-4bd7-8384-40014d49ece7/volumes"
Sep 10 17:30:37.115414 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:37.115402    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba63699-d7b5-4fff-ade7-1afcc4d62684" path="/var/lib/kubelet/pods/fba63699-d7b5-4fff-ade7-1afcc4d62684/volumes"
Sep 10 17:30:42.337298 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337264    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"]
Sep 10 17:30:42.337709 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337689    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba63699-d7b5-4fff-ade7-1afcc4d62684" containerName="main"
Sep 10 17:30:42.337709 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337703    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba63699-d7b5-4fff-ade7-1afcc4d62684" containerName="main"
Sep 10 17:30:42.337786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337725    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff95987c-3c8c-4c2f-b3ab-ead49ca37398" containerName="revoke-keys"
Sep 10 17:30:42.337786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337731    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff95987c-3c8c-4c2f-b3ab-ead49ca37398" containerName="revoke-keys"
Sep 10 17:30:42.337786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337739    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea2f834-65c7-4bd7-8384-40014d49ece7" containerName="main"
Sep 10 17:30:42.337786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337744    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea2f834-65c7-4bd7-8384-40014d49ece7" containerName="main"
Sep 10 17:30:42.337786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337753    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543391cd-d61b-49ce-b014-d4d84288973b" containerName="main"
Sep 10 17:30:42.337786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337758    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="543391cd-d61b-49ce-b014-d4d84288973b" containerName="main"
Sep 10 17:30:42.337959 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337811    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff95987c-3c8c-4c2f-b3ab-ead49ca37398" containerName="revoke-keys"
Sep 10 17:30:42.337959 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337817    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="543391cd-d61b-49ce-b014-d4d84288973b" containerName="main"
Sep 10 17:30:42.337959 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337824    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba63699-d7b5-4fff-ade7-1afcc4d62684" containerName="main"
Sep 10 17:30:42.337959 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.337835    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea2f834-65c7-4bd7-8384-40014d49ece7" containerName="main"
Sep 10 17:30:42.342816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.342800    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.347695 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.347667    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-9e9c14-openshift-default-dockercfg-lwcnw\""
Sep 10 17:30:42.351871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.351853    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:30:42.376750 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.376721    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"]
Sep 10 17:30:42.444210 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444181    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-envoy\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444220    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-token\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444246    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-data\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444261    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kdl\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-kube-api-access-s6kdl\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444290    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-podinfo\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444326    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-certs\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444354    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-socket\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444617 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444381    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-credential-socket\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.444617 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.444410    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istiod-ca-cert\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545295    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-certs\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-socket\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545362    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-credential-socket\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545390    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istiod-ca-cert\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545407    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-envoy\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-token\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545457    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-data\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545822 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545471    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kdl\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-kube-api-access-s6kdl\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545822 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545501    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-podinfo\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545822 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545740    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-socket\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.545822 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545813    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-certs\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.546043 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.545891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-data\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.546102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.546078    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-credential-socket\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.546202 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.546182    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istiod-ca-cert\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.547678 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.547642    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-envoy\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.547965 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.547942    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-podinfo\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.556227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.556203    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-token\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.556986 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.556959    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kdl\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-kube-api-access-s6kdl\") pod \"e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") " pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.653508 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.653436    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:42.937014 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:42.936986    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"]
Sep 10 17:30:42.939552 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:30:42.939507    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa87a685_95c2_4a27_aaf3_0ef4b07a5845.slice/crio-53980f9f2f6a8ae214600781a9e40ac4ccc836ab8c66c17a3595f6dcf4021c9f WatchSource:0}: Error finding container 53980f9f2f6a8ae214600781a9e40ac4ccc836ab8c66c17a3595f6dcf4021c9f: Status 404 returned error can't find the container with id 53980f9f2f6a8ae214600781a9e40ac4ccc836ab8c66c17a3595f6dcf4021c9f
Sep 10 17:30:43.413366 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:43.413322    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"]
Sep 10 17:30:43.418178 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:43.418147    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dcc1cd6c-89lvc"]
Sep 10 17:30:43.734130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:43.734039    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" event={"ID":"aa87a685-95c2-4a27-aaf3-0ef4b07a5845","Type":"ContainerStarted","Data":"53980f9f2f6a8ae214600781a9e40ac4ccc836ab8c66c17a3595f6dcf4021c9f"}
Sep 10 17:30:45.115682 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:45.115642    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff95987c-3c8c-4c2f-b3ab-ead49ca37398" path="/var/lib/kubelet/pods/ff95987c-3c8c-4c2f-b3ab-ead49ca37398/volumes"
Sep 10 17:30:45.368422 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:45.368380    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:30:45.368542 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:45.368456    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:30:45.368542 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:45.368486    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:30:45.745902 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:45.745813    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" event={"ID":"aa87a685-95c2-4a27-aaf3-0ef4b07a5845","Type":"ContainerStarted","Data":"63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da"}
Sep 10 17:30:45.777711 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:45.777663    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" podStartSLOduration=1.3511475179999999 podStartE2EDuration="3.777647115s" totalImagesPullingTime="2.426499597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:45.777566844 +0000 UTC m=+1041.134057237" watchObservedRunningTime="2026-09-10 17:30:45.777647115 +0000 UTC m=+1041.134137514"
Sep 10 17:30:46.654147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:46.654074    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:46.659271 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:46.659246    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:46.750907 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:46.750869    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:46.751704 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:46.751684    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:30:48.690276 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:48.690242    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:30:52.015224 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.015187    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"]
Sep 10 17:30:52.018957 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.018940    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.022671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.022649    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9e9c14-dockercfg-h7kgb\""
Sep 10 17:30:52.023718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.023704    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9e9c14\""
Sep 10 17:30:52.038630 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.038603    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"]
Sep 10 17:30:52.135437 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.135402    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d687446-62ef-4c63-9e0c-e13927f24114-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") " pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.135620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.135461    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxv9\" (UniqueName: \"kubernetes.io/projected/4d687446-62ef-4c63-9e0c-e13927f24114-kube-api-access-bfxv9\") pod \"payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") " pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.236499 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.236465    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d687446-62ef-4c63-9e0c-e13927f24114-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") " pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.236704 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.236521    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxv9\" (UniqueName: \"kubernetes.io/projected/4d687446-62ef-4c63-9e0c-e13927f24114-kube-api-access-bfxv9\") pod \"payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") " pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.237176 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.237155    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d687446-62ef-4c63-9e0c-e13927f24114-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") " pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.246774 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.246745    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxv9\" (UniqueName: \"kubernetes.io/projected/4d687446-62ef-4c63-9e0c-e13927f24114-kube-api-access-bfxv9\") pod \"payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") " pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.328869 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.328836    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.608258 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.608188    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"]
Sep 10 17:30:52.610654 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:30:52.610628    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d687446_62ef_4c63_9e0c_e13927f24114.slice/crio-680d7de7bf32cbc629ab5546ba0e8c6e0a8b39be62475212d2b4abcebbab0bca WatchSource:0}: Error finding container 680d7de7bf32cbc629ab5546ba0e8c6e0a8b39be62475212d2b4abcebbab0bca: Status 404 returned error can't find the container with id 680d7de7bf32cbc629ab5546ba0e8c6e0a8b39be62475212d2b4abcebbab0bca
Sep 10 17:30:52.782182 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.782149    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c" event={"ID":"4d687446-62ef-4c63-9e0c-e13927f24114","Type":"ContainerStarted","Data":"15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f"}
Sep 10 17:30:52.782362 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.782202    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c" event={"ID":"4d687446-62ef-4c63-9e0c-e13927f24114","Type":"ContainerStarted","Data":"680d7de7bf32cbc629ab5546ba0e8c6e0a8b39be62475212d2b4abcebbab0bca"}
Sep 10 17:30:52.782362 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.782310    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:30:52.811817 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:30:52.811774    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c" podStartSLOduration=1.811760064 podStartE2EDuration="1.811760064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:52.811445045 +0000 UTC m=+1048.167935446" watchObservedRunningTime="2026-09-10 17:30:52.811760064 +0000 UTC m=+1048.168250504"
Sep 10 17:31:03.789706 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:03.789675    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:31:04.878742 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:04.878706    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:31:07.857609 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:07.857569    2554 generic.go:356] "Generic (PLEG): container finished" podID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerID="63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da" exitCode=0
Sep 10 17:31:07.858131 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:07.857649    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" event={"ID":"aa87a685-95c2-4a27-aaf3-0ef4b07a5845","Type":"ContainerDied","Data":"63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da"}
Sep 10 17:31:07.858232 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:07.858214    2554 scope.go:117] "RemoveContainer" containerID="63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da"
Sep 10 17:31:07.859333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:07.859306    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:31:07.859399 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:07.859371    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:31:07.859438 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:07.859402    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:31:08.867739 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:08.867701    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" event={"ID":"aa87a685-95c2-4a27-aaf3-0ef4b07a5845","Type":"ContainerStarted","Data":"68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80"}
Sep 10 17:31:09.653957 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:09.653913    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:31:09.654285 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:09.653998    2554 patch_prober.go:28] interesting pod/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 17:31:09.654500 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:09.654449    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 17:31:10.653942 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:10.653901    2554 patch_prober.go:28] interesting pod/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 17:31:10.654962 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:10.654509    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 17:31:11.654500 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:11.654454    2554 patch_prober.go:28] interesting pod/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 17:31:11.654997 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:11.654522    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 17:31:12.654214 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:12.654176    2554 patch_prober.go:28] interesting pod/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 17:31:12.654512 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:12.654483    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 17:31:13.654333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:13.654283    2554 patch_prober.go:28] interesting pod/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 10 17:31:13.654857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:13.654354    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 10 17:31:14.657647 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:14.657613    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:31:14.658051 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:14.657979    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:31:14.658540 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:14.658504    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:31:16.886109 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:16.886075    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:31:17.117440 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.117361    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"]
Sep 10 17:31:17.124683 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.124664    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.134300 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.134276    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15ece2\"/\"test-model-15ece2-kserve-self-signed-certs\""
Sep 10 17:31:17.134523 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.134508    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15ece2\"/\"openshift-service-ca.crt\""
Sep 10 17:31:17.134802 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.134784    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15ece2\"/\"default-dockercfg-trbh5\""
Sep 10 17:31:17.134863 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.134817    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15ece2\"/\"kube-root-ca.crt\""
Sep 10 17:31:17.145861 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.145793    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"]
Sep 10 17:31:17.176953 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.176924    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-dshm\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.177062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.176981    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-tmp-dir\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.177062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.177012    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-model-cache\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.177062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.177030    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-home\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.177062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.177054    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4829976d-6268-4603-9765-793343dead31-tls-certs\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.177211 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.177157    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjfjg\" (UniqueName: \"kubernetes.io/projected/4829976d-6268-4603-9765-793343dead31-kube-api-access-mjfjg\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278217 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278182    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjfjg\" (UniqueName: \"kubernetes.io/projected/4829976d-6268-4603-9765-793343dead31-kube-api-access-mjfjg\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278345 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278267    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-dshm\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278345 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278324    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-tmp-dir\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278481 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278355    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-model-cache\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278481 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278379    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-home\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278481 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4829976d-6268-4603-9765-793343dead31-tls-certs\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278798 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278773    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-model-cache\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.278982 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.278959    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-tmp-dir\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.279340 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.279320    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-home\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.281490 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.281466    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4829976d-6268-4603-9765-793343dead31-tls-certs\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.281976 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.281948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-dshm\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.287815 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.287791    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjfjg\" (UniqueName: \"kubernetes.io/projected/4829976d-6268-4603-9765-793343dead31-kube-api-access-mjfjg\") pod \"test-model-15ece2-kserve-8447fdd96c-qx25q\" (UID: \"4829976d-6268-4603-9765-793343dead31\") " pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.433575 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.433473    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:17.706742 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.706717    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"]
Sep 10 17:31:17.709582 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:31:17.709550    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4829976d_6268_4603_9765_793343dead31.slice/crio-ab71349f4755a198f26b65e439970e003a037dcb3fa00f62dd660398ff25b15c WatchSource:0}: Error finding container ab71349f4755a198f26b65e439970e003a037dcb3fa00f62dd660398ff25b15c: Status 404 returned error can't find the container with id ab71349f4755a198f26b65e439970e003a037dcb3fa00f62dd660398ff25b15c
Sep 10 17:31:17.917871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.917840    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q" event={"ID":"4829976d-6268-4603-9765-793343dead31","Type":"ContainerStarted","Data":"1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6"}
Sep 10 17:31:17.917871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.917879    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q" event={"ID":"4829976d-6268-4603-9765-793343dead31","Type":"ContainerStarted","Data":"ab71349f4755a198f26b65e439970e003a037dcb3fa00f62dd660398ff25b15c"}
Sep 10 17:31:17.940619 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:17.940563    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q" podStartSLOduration=0.940546944 podStartE2EDuration="940.546944ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:17.938670699 +0000 UTC m=+1073.295161112" watchObservedRunningTime="2026-09-10 17:31:17.940546944 +0000 UTC m=+1073.297037341"
Sep 10 17:31:27.433754 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:27.433719    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:27.434121 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:27.433782    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:27.446400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:27.446372    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:27.683984 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:27.683906    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:31:27.981509 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:27.981419    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:31:42.279415 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:31:42.279375    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:32:10.286000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:10.285963    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:32:22.689143 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:22.689085    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"]
Sep 10 17:32:22.689649 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:22.689359    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c" podUID="4d687446-62ef-4c63-9e0c-e13927f24114" containerName="payload-processing" containerID="cri-o://15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f" gracePeriod=30
Sep 10 17:32:22.935812 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:22.935790    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:32:23.110854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.110821    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfxv9\" (UniqueName: \"kubernetes.io/projected/4d687446-62ef-4c63-9e0c-e13927f24114-kube-api-access-bfxv9\") pod \"4d687446-62ef-4c63-9e0c-e13927f24114\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") "
Sep 10 17:32:23.111053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.110910    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d687446-62ef-4c63-9e0c-e13927f24114-plugins-config-volume\") pod \"4d687446-62ef-4c63-9e0c-e13927f24114\" (UID: \"4d687446-62ef-4c63-9e0c-e13927f24114\") "
Sep 10 17:32:23.111289 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.111261    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d687446-62ef-4c63-9e0c-e13927f24114-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d687446-62ef-4c63-9e0c-e13927f24114" (UID: "4d687446-62ef-4c63-9e0c-e13927f24114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:32:23.113163 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.113133    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d687446-62ef-4c63-9e0c-e13927f24114-kube-api-access-bfxv9" (OuterVolumeSpecName: "kube-api-access-bfxv9") pod "4d687446-62ef-4c63-9e0c-e13927f24114" (UID: "4d687446-62ef-4c63-9e0c-e13927f24114"). InnerVolumeSpecName "kube-api-access-bfxv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:23.212365 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.212331    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxv9\" (UniqueName: \"kubernetes.io/projected/4d687446-62ef-4c63-9e0c-e13927f24114-kube-api-access-bfxv9\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:23.212365 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.212366    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d687446-62ef-4c63-9e0c-e13927f24114-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:23.239368 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.239341    2554 generic.go:356] "Generic (PLEG): container finished" podID="4d687446-62ef-4c63-9e0c-e13927f24114" containerID="15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f" exitCode=0
Sep 10 17:32:23.239475 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.239390    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"
Sep 10 17:32:23.239475 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.239387    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c" event={"ID":"4d687446-62ef-4c63-9e0c-e13927f24114","Type":"ContainerDied","Data":"15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f"}
Sep 10 17:32:23.239475 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.239449    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c" event={"ID":"4d687446-62ef-4c63-9e0c-e13927f24114","Type":"ContainerDied","Data":"680d7de7bf32cbc629ab5546ba0e8c6e0a8b39be62475212d2b4abcebbab0bca"}
Sep 10 17:32:23.239475 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.239473    2554 scope.go:117] "RemoveContainer" containerID="15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f"
Sep 10 17:32:23.248199 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.248178    2554 scope.go:117] "RemoveContainer" containerID="15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f"
Sep 10 17:32:23.248464 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:32:23.248446    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f\": container with ID starting with 15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f not found: ID does not exist" containerID="15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f"
Sep 10 17:32:23.248518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.248472    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f"} err="failed to get container status \"15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f\": rpc error: code = NotFound desc = could not find container \"15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f\": container with ID starting with 15441fbc0aab38380973ebe58103a46dac4851db233411d0e0926e62716bfc9f not found: ID does not exist"
Sep 10 17:32:23.257728 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.257706    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"]
Sep 10 17:32:23.261892 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:23.261872    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9e9c14-589578dcc-kt84c"]
Sep 10 17:32:25.114610 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:25.114573    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d687446-62ef-4c63-9e0c-e13927f24114" path="/var/lib/kubelet/pods/4d687446-62ef-4c63-9e0c-e13927f24114/volumes"
Sep 10 17:32:30.070722 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:30.070686    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:32:30.882817 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:30.882777    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"]
Sep 10 17:32:30.883041 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:30.883003    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy" containerID="cri-o://68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80" gracePeriod=30
Sep 10 17:32:31.739416 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.739379    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"]
Sep 10 17:32:31.739832 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.739764    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d687446-62ef-4c63-9e0c-e13927f24114" containerName="payload-processing"
Sep 10 17:32:31.739832 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.739777    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d687446-62ef-4c63-9e0c-e13927f24114" containerName="payload-processing"
Sep 10 17:32:31.739908 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.739834    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d687446-62ef-4c63-9e0c-e13927f24114" containerName="payload-processing"
Sep 10 17:32:31.743897 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.743880    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:31.747275 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.747255    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-851d1193\""
Sep 10 17:32:31.747374 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.747298    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-851d1193-dockercfg-vl5nq\""
Sep 10 17:32:31.755871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.755848    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"]
Sep 10 17:32:31.793884 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.793840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca6d684-11f1-4295-bebe-f12b9c3e5882-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:31.794034 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.793960    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzcg\" (UniqueName: \"kubernetes.io/projected/dca6d684-11f1-4295-bebe-f12b9c3e5882-kube-api-access-rvzcg\") pod \"payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:31.895209 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.895174    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzcg\" (UniqueName: \"kubernetes.io/projected/dca6d684-11f1-4295-bebe-f12b9c3e5882-kube-api-access-rvzcg\") pod \"payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:31.895379 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.895229    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca6d684-11f1-4295-bebe-f12b9c3e5882-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:31.895898 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.895879    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca6d684-11f1-4295-bebe-f12b9c3e5882-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:31.903755 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:31.903733    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzcg\" (UniqueName: \"kubernetes.io/projected/dca6d684-11f1-4295-bebe-f12b9c3e5882-kube-api-access-rvzcg\") pod \"payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:32.053947 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:32.053914    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:32.325576 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:32.325553    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"]
Sep 10 17:32:32.327594 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:32:32.327563    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca6d684_11f1_4295_bebe_f12b9c3e5882.slice/crio-086917fc6adc0519b83ba95a115b78429edd0bd1f453151c454a4992b602a0a4 WatchSource:0}: Error finding container 086917fc6adc0519b83ba95a115b78429edd0bd1f453151c454a4992b602a0a4: Status 404 returned error can't find the container with id 086917fc6adc0519b83ba95a115b78429edd0bd1f453151c454a4992b602a0a4
Sep 10 17:32:33.293215 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:33.293178    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh" event={"ID":"dca6d684-11f1-4295-bebe-f12b9c3e5882","Type":"ContainerStarted","Data":"f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906"}
Sep 10 17:32:33.293622 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:33.293239    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh" event={"ID":"dca6d684-11f1-4295-bebe-f12b9c3e5882","Type":"ContainerStarted","Data":"086917fc6adc0519b83ba95a115b78429edd0bd1f453151c454a4992b602a0a4"}
Sep 10 17:32:33.293622 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:33.293266    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:33.320639 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:33.320591    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh" podStartSLOduration=2.320574903 podStartE2EDuration="2.320574903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:33.319706882 +0000 UTC m=+1148.676197282" watchObservedRunningTime="2026-09-10 17:32:33.320574903 +0000 UTC m=+1148.677065316"
Sep 10 17:32:36.127403 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.127380    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:32:36.237079 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237002    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-envoy\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237079 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237054    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-credential-socket\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237086    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istiod-ca-cert\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237103    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-podinfo\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237122    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-token\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237138    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-data\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237168    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-socket\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237239    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-certs\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237267    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6kdl\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-kube-api-access-s6kdl\") pod \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\" (UID: \"aa87a685-95c2-4a27-aaf3-0ef4b07a5845\") "
Sep 10 17:32:36.237700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237402    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:32:36.237700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237576    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:32:36.237700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237673    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:32:36.237868 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237758    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:32:36.237868 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.237576    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:32:36.238082 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.238064    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.238124 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.238089    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-credential-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.238124 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.238103    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istiod-ca-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.238124 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.238118    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-data\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.238238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.238131    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-workload-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.239632 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.239612    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:32:36.239747 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.239723    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-kube-api-access-s6kdl" (OuterVolumeSpecName: "kube-api-access-s6kdl") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "kube-api-access-s6kdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:36.239806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.239772    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:32:36.239806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.239785    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa87a685-95c2-4a27-aaf3-0ef4b07a5845" (UID: "aa87a685-95c2-4a27-aaf3-0ef4b07a5845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:32:36.315126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.315087    2554 generic.go:356] "Generic (PLEG): container finished" podID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerID="68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80" exitCode=0
Sep 10 17:32:36.315297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.315154    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" event={"ID":"aa87a685-95c2-4a27-aaf3-0ef4b07a5845","Type":"ContainerDied","Data":"68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80"}
Sep 10 17:32:36.315297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.315161    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"
Sep 10 17:32:36.315297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.315236    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9" event={"ID":"aa87a685-95c2-4a27-aaf3-0ef4b07a5845","Type":"ContainerDied","Data":"53980f9f2f6a8ae214600781a9e40ac4ccc836ab8c66c17a3595f6dcf4021c9f"}
Sep 10 17:32:36.315297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.315266    2554 scope.go:117] "RemoveContainer" containerID="68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80"
Sep 10 17:32:36.324839 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.324824    2554 scope.go:117] "RemoveContainer" containerID="63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da"
Sep 10 17:32:36.332786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.332768    2554 scope.go:117] "RemoveContainer" containerID="68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80"
Sep 10 17:32:36.333050 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:32:36.333033    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80\": container with ID starting with 68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80 not found: ID does not exist" containerID="68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80"
Sep 10 17:32:36.333098 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.333058    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80"} err="failed to get container status \"68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80\": rpc error: code = NotFound desc = could not find container \"68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80\": container with ID starting with 68864e8e0159445b8a16c5be51543b2ebab383f8f8e9110d206f6bd7a9c05b80 not found: ID does not exist"
Sep 10 17:32:36.333098 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.333072    2554 scope.go:117] "RemoveContainer" containerID="63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da"
Sep 10 17:32:36.333307 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:32:36.333288    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da\": container with ID starting with 63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da not found: ID does not exist" containerID="63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da"
Sep 10 17:32:36.333344 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.333312    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da"} err="failed to get container status \"63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da\": rpc error: code = NotFound desc = could not find container \"63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da\": container with ID starting with 63859f4d9fd9769f9f420d66507088318ffd26418b8b4d4edd33ff292d2da9da not found: ID does not exist"
Sep 10 17:32:36.338819 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.338801    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6kdl\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-kube-api-access-s6kdl\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.338883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.338821    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-envoy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.338883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.338830    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-podinfo\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.338883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.338838    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa87a685-95c2-4a27-aaf3-0ef4b07a5845-istio-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:32:36.345673 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.345653    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"]
Sep 10 17:32:36.355884 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:36.355864    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-9e9c14-openshift-default-6d6f9857cb-5vkl9"]
Sep 10 17:32:37.115297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:37.115254    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" path="/var/lib/kubelet/pods/aa87a685-95c2-4a27-aaf3-0ef4b07a5845/volumes"
Sep 10 17:32:43.886918 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:43.886879    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:32:44.301205 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:44.301158    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:32:55.319307 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.319224    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"]
Sep 10 17:32:55.319819 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.319750    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy"
Sep 10 17:32:55.319819 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.319771    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy"
Sep 10 17:32:55.319935 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.319892    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy"
Sep 10 17:32:55.319935 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.319905    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy"
Sep 10 17:32:55.323060 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.323038    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.325864 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.325837    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8bd101\"/\"kube-root-ca.crt\""
Sep 10 17:32:55.327088 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.327065    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8bd101\"/\"noten-model-8bd101-kserve-self-signed-certs\""
Sep 10 17:32:55.327088 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.327083    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8bd101\"/\"openshift-service-ca.crt\""
Sep 10 17:32:55.327279 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.327085    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8bd101\"/\"default-dockercfg-q4gsc\""
Sep 10 17:32:55.335869 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.335845    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"]
Sep 10 17:32:55.420717 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.420677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-dshm\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.420903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.420730    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tmp-dir\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.420903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.420762    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnkwn\" (UniqueName: \"kubernetes.io/projected/81895ac3-7b87-4ac8-af51-cb9af0d6291e-kube-api-access-pnkwn\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.420903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.420780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tls-certs\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.420903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.420807    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-model-cache\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.420903 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.420897    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-home\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521545 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tmp-dir\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521708 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521567    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnkwn\" (UniqueName: \"kubernetes.io/projected/81895ac3-7b87-4ac8-af51-cb9af0d6291e-kube-api-access-pnkwn\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521708 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521588    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tls-certs\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521708 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-model-cache\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521708 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521658    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-home\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521729    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-dshm\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521947    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tmp-dir\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.521983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.521970    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-model-cache\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.522082 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.522056    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-home\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.523898 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.523878    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-dshm\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.524227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.524212    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tls-certs\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.530518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.530498    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnkwn\" (UniqueName: \"kubernetes.io/projected/81895ac3-7b87-4ac8-af51-cb9af0d6291e-kube-api-access-pnkwn\") pod \"noten-model-8bd101-kserve-68cb867f9f-6xvzb\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") " pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.634414 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.634335    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:32:55.903938 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:55.903912    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"]
Sep 10 17:32:55.905315 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:32:55.905291    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81895ac3_7b87_4ac8_af51_cb9af0d6291e.slice/crio-7b670f469be507f5783e1140cacd343a46f020290a4213a47f4642c057680e7e WatchSource:0}: Error finding container 7b670f469be507f5783e1140cacd343a46f020290a4213a47f4642c057680e7e: Status 404 returned error can't find the container with id 7b670f469be507f5783e1140cacd343a46f020290a4213a47f4642c057680e7e
Sep 10 17:32:56.417580 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:56.417543    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb" event={"ID":"81895ac3-7b87-4ac8-af51-cb9af0d6291e","Type":"ContainerStarted","Data":"8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01"}
Sep 10 17:32:56.418005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:56.417588    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb" event={"ID":"81895ac3-7b87-4ac8-af51-cb9af0d6291e","Type":"ContainerStarted","Data":"7b670f469be507f5783e1140cacd343a46f020290a4213a47f4642c057680e7e"}
Sep 10 17:32:56.440046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:32:56.439999    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb" podStartSLOduration=1.439980576 podStartE2EDuration="1.439980576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:56.438156624 +0000 UTC m=+1171.794647026" watchObservedRunningTime="2026-09-10 17:32:56.439980576 +0000 UTC m=+1171.796470978"
Sep 10 17:33:05.634718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:05.634681    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:33:05.635193 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:05.634734    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:33:05.647061 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:05.647035    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:33:06.476504 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:06.476477    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:33:06.885658 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:06.885624    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:33:11.496985 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.496955    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"]
Sep 10 17:33:11.497464 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.497197    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh" podUID="dca6d684-11f1-4295-bebe-f12b9c3e5882" containerName="payload-pre-processing" containerID="cri-o://f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906" gracePeriod=30
Sep 10 17:33:11.742295 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.742271    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:33:11.778127 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.778046    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvzcg\" (UniqueName: \"kubernetes.io/projected/dca6d684-11f1-4295-bebe-f12b9c3e5882-kube-api-access-rvzcg\") pod \"dca6d684-11f1-4295-bebe-f12b9c3e5882\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") "
Sep 10 17:33:11.778288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.778181    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca6d684-11f1-4295-bebe-f12b9c3e5882-plugins-config-volume\") pod \"dca6d684-11f1-4295-bebe-f12b9c3e5882\" (UID: \"dca6d684-11f1-4295-bebe-f12b9c3e5882\") "
Sep 10 17:33:11.778881 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.778855    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca6d684-11f1-4295-bebe-f12b9c3e5882-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dca6d684-11f1-4295-bebe-f12b9c3e5882" (UID: "dca6d684-11f1-4295-bebe-f12b9c3e5882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:33:11.780815 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.780793    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca6d684-11f1-4295-bebe-f12b9c3e5882-kube-api-access-rvzcg" (OuterVolumeSpecName: "kube-api-access-rvzcg") pod "dca6d684-11f1-4295-bebe-f12b9c3e5882" (UID: "dca6d684-11f1-4295-bebe-f12b9c3e5882"). InnerVolumeSpecName "kube-api-access-rvzcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:11.879787 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.879752    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dca6d684-11f1-4295-bebe-f12b9c3e5882-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:11.879787 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:11.879782    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzcg\" (UniqueName: \"kubernetes.io/projected/dca6d684-11f1-4295-bebe-f12b9c3e5882-kube-api-access-rvzcg\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:12.496656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.496618    2554 generic.go:356] "Generic (PLEG): container finished" podID="dca6d684-11f1-4295-bebe-f12b9c3e5882" containerID="f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906" exitCode=0
Sep 10 17:33:12.496854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.496692    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"
Sep 10 17:33:12.496854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.496704    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh" event={"ID":"dca6d684-11f1-4295-bebe-f12b9c3e5882","Type":"ContainerDied","Data":"f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906"}
Sep 10 17:33:12.496854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.496762    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh" event={"ID":"dca6d684-11f1-4295-bebe-f12b9c3e5882","Type":"ContainerDied","Data":"086917fc6adc0519b83ba95a115b78429edd0bd1f453151c454a4992b602a0a4"}
Sep 10 17:33:12.496854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.496785    2554 scope.go:117] "RemoveContainer" containerID="f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906"
Sep 10 17:33:12.506132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.506115    2554 scope.go:117] "RemoveContainer" containerID="f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906"
Sep 10 17:33:12.506415 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:33:12.506396    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906\": container with ID starting with f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906 not found: ID does not exist" containerID="f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906"
Sep 10 17:33:12.506463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.506423    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906"} err="failed to get container status \"f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906\": rpc error: code = NotFound desc = could not find container \"f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906\": container with ID starting with f5126bf30e1e89b0dae0815ff755a97fe149864fb5534595e7677ace886f6906 not found: ID does not exist"
Sep 10 17:33:12.518778 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.518751    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"]
Sep 10 17:33:12.523790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:12.523768    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-851d1193-fd667654b-zqmgh"]
Sep 10 17:33:13.114898 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:13.114852    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dca6d684-11f1-4295-bebe-f12b9c3e5882" path="/var/lib/kubelet/pods/dca6d684-11f1-4295-bebe-f12b9c3e5882/volumes"
Sep 10 17:33:15.593484 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:15.593453    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"]
Sep 10 17:33:16.582313 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:16.582279    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"]
Sep 10 17:33:17.687545 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:17.687494    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"]
Sep 10 17:33:18.783051 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:18.783017    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:33:25.082426 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:25.082398    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:33:25.083567 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:25.083546    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:33:28.870807 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:28.870764    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:33:30.613254 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.613211    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls" podUID="14955e52-c8a8-40b7-91e0-55421e9fa3a6" containerName="main" containerID="cri-o://958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f" gracePeriod=45
Sep 10 17:33:30.867129 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.867071    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:33:30.953204 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953170    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-home\") pod \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") "
Sep 10 17:33:30.953204 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953206    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tls-certs\") pod \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") "
Sep 10 17:33:30.953443 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953227    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tmp-dir\") pod \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") "
Sep 10 17:33:30.953443 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953264    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zvr9\" (UniqueName: \"kubernetes.io/projected/14955e52-c8a8-40b7-91e0-55421e9fa3a6-kube-api-access-8zvr9\") pod \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") "
Sep 10 17:33:30.953443 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953284    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-dshm\") pod \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") "
Sep 10 17:33:30.953617 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953475    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-model-cache\") pod \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\" (UID: \"14955e52-c8a8-40b7-91e0-55421e9fa3a6\") "
Sep 10 17:33:30.953617 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953518    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-home" (OuterVolumeSpecName: "home") pod "14955e52-c8a8-40b7-91e0-55421e9fa3a6" (UID: "14955e52-c8a8-40b7-91e0-55421e9fa3a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:30.953617 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953592    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14955e52-c8a8-40b7-91e0-55421e9fa3a6" (UID: "14955e52-c8a8-40b7-91e0-55421e9fa3a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:30.953779 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953727    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "14955e52-c8a8-40b7-91e0-55421e9fa3a6" (UID: "14955e52-c8a8-40b7-91e0-55421e9fa3a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:30.953943 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953919    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:30.954054 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953947    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:30.954054 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.953962    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:30.955567 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.955523    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14955e52-c8a8-40b7-91e0-55421e9fa3a6" (UID: "14955e52-c8a8-40b7-91e0-55421e9fa3a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:30.955567 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.955558    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-dshm" (OuterVolumeSpecName: "dshm") pod "14955e52-c8a8-40b7-91e0-55421e9fa3a6" (UID: "14955e52-c8a8-40b7-91e0-55421e9fa3a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:30.955698 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:30.955618    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14955e52-c8a8-40b7-91e0-55421e9fa3a6-kube-api-access-8zvr9" (OuterVolumeSpecName: "kube-api-access-8zvr9") pod "14955e52-c8a8-40b7-91e0-55421e9fa3a6" (UID: "14955e52-c8a8-40b7-91e0-55421e9fa3a6"). InnerVolumeSpecName "kube-api-access-8zvr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:31.054419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.054381    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14955e52-c8a8-40b7-91e0-55421e9fa3a6-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:31.054419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.054413    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvr9\" (UniqueName: \"kubernetes.io/projected/14955e52-c8a8-40b7-91e0-55421e9fa3a6-kube-api-access-8zvr9\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:31.054419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.054425    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14955e52-c8a8-40b7-91e0-55421e9fa3a6-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:31.593792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.593758    2554 generic.go:356] "Generic (PLEG): container finished" podID="14955e52-c8a8-40b7-91e0-55421e9fa3a6" containerID="958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f" exitCode=0
Sep 10 17:33:31.593966 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.593801    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"
Sep 10 17:33:31.593966 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.593830    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls" event={"ID":"14955e52-c8a8-40b7-91e0-55421e9fa3a6","Type":"ContainerDied","Data":"958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f"}
Sep 10 17:33:31.593966 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.593868    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls" event={"ID":"14955e52-c8a8-40b7-91e0-55421e9fa3a6","Type":"ContainerDied","Data":"c9eaf16e50087d4ca4d8704876cdc14ed289f85ffe6969819b3ded032eadfa80"}
Sep 10 17:33:31.593966 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.593885    2554 scope.go:117] "RemoveContainer" containerID="958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f"
Sep 10 17:33:31.602366 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.602346    2554 scope.go:117] "RemoveContainer" containerID="958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f"
Sep 10 17:33:31.602552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.602499    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb" podUID="81895ac3-7b87-4ac8-af51-cb9af0d6291e" containerName="main" containerID="cri-o://8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01" gracePeriod=45
Sep 10 17:33:31.602672 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:33:31.602652    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f\": container with ID starting with 958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f not found: ID does not exist" containerID="958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f"
Sep 10 17:33:31.602730 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.602680    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f"} err="failed to get container status \"958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f\": rpc error: code = NotFound desc = could not find container \"958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f\": container with ID starting with 958a807ecce14acc3b7968c04c88b7d454096e62dea486e0123e655acdf5831f not found: ID does not exist"
Sep 10 17:33:31.612055 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.612035    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"]
Sep 10 17:33:31.619076 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.619054    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8c46e8/test-model-8c46e8-kserve-b5ccbf9f7-99fls"]
Sep 10 17:33:31.847113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.847054    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:33:31.962572 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.962518    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tmp-dir\") pod \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") "
Sep 10 17:33:31.962737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.962579    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-dshm\") pod \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") "
Sep 10 17:33:31.962737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.962689    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tls-certs\") pod \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") "
Sep 10 17:33:31.962737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.962715    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnkwn\" (UniqueName: \"kubernetes.io/projected/81895ac3-7b87-4ac8-af51-cb9af0d6291e-kube-api-access-pnkwn\") pod \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") "
Sep 10 17:33:31.962924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.962771    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-model-cache\") pod \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") "
Sep 10 17:33:31.962924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.962814    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81895ac3-7b87-4ac8-af51-cb9af0d6291e" (UID: "81895ac3-7b87-4ac8-af51-cb9af0d6291e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:31.962924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.962831    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-home\") pod \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\" (UID: \"81895ac3-7b87-4ac8-af51-cb9af0d6291e\") "
Sep 10 17:33:31.963126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.963098    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-home" (OuterVolumeSpecName: "home") pod "81895ac3-7b87-4ac8-af51-cb9af0d6291e" (UID: "81895ac3-7b87-4ac8-af51-cb9af0d6291e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:31.963126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.963113    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-model-cache" (OuterVolumeSpecName: "model-cache") pod "81895ac3-7b87-4ac8-af51-cb9af0d6291e" (UID: "81895ac3-7b87-4ac8-af51-cb9af0d6291e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:31.963247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.963168    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:31.964901 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.964875    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81895ac3-7b87-4ac8-af51-cb9af0d6291e" (UID: "81895ac3-7b87-4ac8-af51-cb9af0d6291e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:31.965005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.964880    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-dshm" (OuterVolumeSpecName: "dshm") pod "81895ac3-7b87-4ac8-af51-cb9af0d6291e" (UID: "81895ac3-7b87-4ac8-af51-cb9af0d6291e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:31.965005 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:31.964953    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81895ac3-7b87-4ac8-af51-cb9af0d6291e-kube-api-access-pnkwn" (OuterVolumeSpecName: "kube-api-access-pnkwn") pod "81895ac3-7b87-4ac8-af51-cb9af0d6291e" (UID: "81895ac3-7b87-4ac8-af51-cb9af0d6291e"). InnerVolumeSpecName "kube-api-access-pnkwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:32.064706 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.064675    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81895ac3-7b87-4ac8-af51-cb9af0d6291e-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:32.064706 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.064701    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnkwn\" (UniqueName: \"kubernetes.io/projected/81895ac3-7b87-4ac8-af51-cb9af0d6291e-kube-api-access-pnkwn\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:32.064706 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.064711    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:32.064924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.064722    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:32.064924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.064729    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81895ac3-7b87-4ac8-af51-cb9af0d6291e-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:32.601574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.601520    2554 generic.go:356] "Generic (PLEG): container finished" podID="81895ac3-7b87-4ac8-af51-cb9af0d6291e" containerID="8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01" exitCode=0
Sep 10 17:33:32.601574 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.601562    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb" event={"ID":"81895ac3-7b87-4ac8-af51-cb9af0d6291e","Type":"ContainerDied","Data":"8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01"}
Sep 10 17:33:32.601804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.601596    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"
Sep 10 17:33:32.601804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.601618    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb" event={"ID":"81895ac3-7b87-4ac8-af51-cb9af0d6291e","Type":"ContainerDied","Data":"7b670f469be507f5783e1140cacd343a46f020290a4213a47f4642c057680e7e"}
Sep 10 17:33:32.601804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.601639    2554 scope.go:117] "RemoveContainer" containerID="8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01"
Sep 10 17:33:32.611023 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.610999    2554 scope.go:117] "RemoveContainer" containerID="8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01"
Sep 10 17:33:32.611310 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:33:32.611278    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01\": container with ID starting with 8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01 not found: ID does not exist" containerID="8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01"
Sep 10 17:33:32.611405 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.611318    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01"} err="failed to get container status \"8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01\": rpc error: code = NotFound desc = could not find container \"8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01\": container with ID starting with 8f25f72490ae436de3006e4a00f0c5c9ce5ceef7b0a80a816fd3ef5afdd6bd01 not found: ID does not exist"
Sep 10 17:33:32.626805 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.626781    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"]
Sep 10 17:33:32.633201 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.633181    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-8bd101/noten-model-8bd101-kserve-68cb867f9f-6xvzb"]
Sep 10 17:33:32.707006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.706956    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q" podUID="4829976d-6268-4603-9765-793343dead31" containerName="main" containerID="cri-o://1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6" gracePeriod=45
Sep 10 17:33:32.951138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:32.951114    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:33:33.074656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.074619    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4829976d-6268-4603-9765-793343dead31-tls-certs\") pod \"4829976d-6268-4603-9765-793343dead31\" (UID: \"4829976d-6268-4603-9765-793343dead31\") "
Sep 10 17:33:33.074825 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.074675    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-model-cache\") pod \"4829976d-6268-4603-9765-793343dead31\" (UID: \"4829976d-6268-4603-9765-793343dead31\") "
Sep 10 17:33:33.074825 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.074720    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-home\") pod \"4829976d-6268-4603-9765-793343dead31\" (UID: \"4829976d-6268-4603-9765-793343dead31\") "
Sep 10 17:33:33.074925 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.074817    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-tmp-dir\") pod \"4829976d-6268-4603-9765-793343dead31\" (UID: \"4829976d-6268-4603-9765-793343dead31\") "
Sep 10 17:33:33.074925 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.074864    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjfjg\" (UniqueName: \"kubernetes.io/projected/4829976d-6268-4603-9765-793343dead31-kube-api-access-mjfjg\") pod \"4829976d-6268-4603-9765-793343dead31\" (UID: \"4829976d-6268-4603-9765-793343dead31\") "
Sep 10 17:33:33.074925 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.074905    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-dshm\") pod \"4829976d-6268-4603-9765-793343dead31\" (UID: \"4829976d-6268-4603-9765-793343dead31\") "
Sep 10 17:33:33.075092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.074947    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-home" (OuterVolumeSpecName: "home") pod "4829976d-6268-4603-9765-793343dead31" (UID: "4829976d-6268-4603-9765-793343dead31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:33.075092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.075017    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-model-cache" (OuterVolumeSpecName: "model-cache") pod "4829976d-6268-4603-9765-793343dead31" (UID: "4829976d-6268-4603-9765-793343dead31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:33.075092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.075065    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4829976d-6268-4603-9765-793343dead31" (UID: "4829976d-6268-4603-9765-793343dead31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:33.075266 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.075246    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:33.075305 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.075270    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:33.075305 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.075285    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:33.076974 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.076945    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-dshm" (OuterVolumeSpecName: "dshm") pod "4829976d-6268-4603-9765-793343dead31" (UID: "4829976d-6268-4603-9765-793343dead31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:33.077090 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.077064    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4829976d-6268-4603-9765-793343dead31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4829976d-6268-4603-9765-793343dead31" (UID: "4829976d-6268-4603-9765-793343dead31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:33.077144 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.077094    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4829976d-6268-4603-9765-793343dead31-kube-api-access-mjfjg" (OuterVolumeSpecName: "kube-api-access-mjfjg") pod "4829976d-6268-4603-9765-793343dead31" (UID: "4829976d-6268-4603-9765-793343dead31"). InnerVolumeSpecName "kube-api-access-mjfjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:33.115458 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.115427    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14955e52-c8a8-40b7-91e0-55421e9fa3a6" path="/var/lib/kubelet/pods/14955e52-c8a8-40b7-91e0-55421e9fa3a6/volumes"
Sep 10 17:33:33.115876 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.115861    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81895ac3-7b87-4ac8-af51-cb9af0d6291e" path="/var/lib/kubelet/pods/81895ac3-7b87-4ac8-af51-cb9af0d6291e/volumes"
Sep 10 17:33:33.176345 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.176277    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjfjg\" (UniqueName: \"kubernetes.io/projected/4829976d-6268-4603-9765-793343dead31-kube-api-access-mjfjg\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:33.176345 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.176302    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4829976d-6268-4603-9765-793343dead31-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:33.176345 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.176313    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4829976d-6268-4603-9765-793343dead31-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:33.609457 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.609417    2554 generic.go:356] "Generic (PLEG): container finished" podID="4829976d-6268-4603-9765-793343dead31" containerID="1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6" exitCode=0
Sep 10 17:33:33.609641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.609463    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"
Sep 10 17:33:33.609641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.609478    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q" event={"ID":"4829976d-6268-4603-9765-793343dead31","Type":"ContainerDied","Data":"1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6"}
Sep 10 17:33:33.609641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.609514    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q" event={"ID":"4829976d-6268-4603-9765-793343dead31","Type":"ContainerDied","Data":"ab71349f4755a198f26b65e439970e003a037dcb3fa00f62dd660398ff25b15c"}
Sep 10 17:33:33.609641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.609553    2554 scope.go:117] "RemoveContainer" containerID="1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6"
Sep 10 17:33:33.617804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.617785    2554 scope.go:117] "RemoveContainer" containerID="1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6"
Sep 10 17:33:33.618071 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:33:33.618051    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6\": container with ID starting with 1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6 not found: ID does not exist" containerID="1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6"
Sep 10 17:33:33.618111 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.618080    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6"} err="failed to get container status \"1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6\": rpc error: code = NotFound desc = could not find container \"1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6\": container with ID starting with 1c8941727ffc9cf4755284b56174eea862845cfc326df144a91fd1323d540cb6 not found: ID does not exist"
Sep 10 17:33:33.629074 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.629048    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"]
Sep 10 17:33:33.632808 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:33.632787    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-15ece2/test-model-15ece2-kserve-8447fdd96c-qx25q"]
Sep 10 17:33:35.114519 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:35.114473    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4829976d-6268-4603-9765-793343dead31" path="/var/lib/kubelet/pods/4829976d-6268-4603-9765-793343dead31/volumes"
Sep 10 17:33:36.899766 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:36.899730    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:33:47.239796 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.239763    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"]
Sep 10 17:33:47.240312 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.239964    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q" podUID="68ddde51-c250-49d2-a931-b43a53956de2" containerName="maas-api" containerID="cri-o://887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f" gracePeriod=30
Sep 10 17:33:47.299410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.299377    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"]
Sep 10 17:33:47.306052 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.306028    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-15ece2-29817690-4n9bt"]
Sep 10 17:33:47.486946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.486920    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:33:47.502936 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.502876    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddde51-c250-49d2-a931-b43a53956de2-openshift-service-ca\") pod \"68ddde51-c250-49d2-a931-b43a53956de2\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") "
Sep 10 17:33:47.503065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.502949    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-metrics-certs\") pod \"68ddde51-c250-49d2-a931-b43a53956de2\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") "
Sep 10 17:33:47.503065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.503003    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls\") pod \"68ddde51-c250-49d2-a931-b43a53956de2\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") "
Sep 10 17:33:47.503065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.503062    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhbrv\" (UniqueName: \"kubernetes.io/projected/68ddde51-c250-49d2-a931-b43a53956de2-kube-api-access-zhbrv\") pod \"68ddde51-c250-49d2-a931-b43a53956de2\" (UID: \"68ddde51-c250-49d2-a931-b43a53956de2\") "
Sep 10 17:33:47.503283 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.503250    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ddde51-c250-49d2-a931-b43a53956de2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68ddde51-c250-49d2-a931-b43a53956de2" (UID: "68ddde51-c250-49d2-a931-b43a53956de2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:33:47.503417 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.503366    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ddde51-c250-49d2-a931-b43a53956de2-openshift-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:47.505232 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.505202    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68ddde51-c250-49d2-a931-b43a53956de2" (UID: "68ddde51-c250-49d2-a931-b43a53956de2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:47.505318 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.505230    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68ddde51-c250-49d2-a931-b43a53956de2" (UID: "68ddde51-c250-49d2-a931-b43a53956de2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:47.505318 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.505265    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ddde51-c250-49d2-a931-b43a53956de2-kube-api-access-zhbrv" (OuterVolumeSpecName: "kube-api-access-zhbrv") pod "68ddde51-c250-49d2-a931-b43a53956de2" (UID: "68ddde51-c250-49d2-a931-b43a53956de2"). InnerVolumeSpecName "kube-api-access-zhbrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:47.604496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.604458    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-maas-api-tls\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:47.604496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.604489    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhbrv\" (UniqueName: \"kubernetes.io/projected/68ddde51-c250-49d2-a931-b43a53956de2-kube-api-access-zhbrv\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:47.604496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.604500    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ddde51-c250-49d2-a931-b43a53956de2-metrics-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:33:47.681867 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.681833    2554 generic.go:356] "Generic (PLEG): container finished" podID="68ddde51-c250-49d2-a931-b43a53956de2" containerID="887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f" exitCode=0
Sep 10 17:33:47.682050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.681879    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"
Sep 10 17:33:47.682050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.681919    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q" event={"ID":"68ddde51-c250-49d2-a931-b43a53956de2","Type":"ContainerDied","Data":"887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f"}
Sep 10 17:33:47.682050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.681974    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q" event={"ID":"68ddde51-c250-49d2-a931-b43a53956de2","Type":"ContainerDied","Data":"f19a5d1fc022ab2f13a4d30c346bf5e725fb619e08f615dbbe8003a093726b41"}
Sep 10 17:33:47.682050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.681996    2554 scope.go:117] "RemoveContainer" containerID="887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f"
Sep 10 17:33:47.692688 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.692672    2554 scope.go:117] "RemoveContainer" containerID="887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f"
Sep 10 17:33:47.692933 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:33:47.692914    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f\": container with ID starting with 887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f not found: ID does not exist" containerID="887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f"
Sep 10 17:33:47.693007 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.692946    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f"} err="failed to get container status \"887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f\": rpc error: code = NotFound desc = could not find container \"887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f\": container with ID starting with 887dea8ad6d5317d35640a3196a6c575d69de9cdfd7b7e9f498c5003453bfe2f not found: ID does not exist"
Sep 10 17:33:47.720751 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.720720    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"]
Sep 10 17:33:47.733024 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:47.732993    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15ece2-85d4477fbc-q7n7q"]
Sep 10 17:33:49.114459 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:49.114425    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d97b3a0-a60a-45d7-85a2-1debe9a16a65" path="/var/lib/kubelet/pods/1d97b3a0-a60a-45d7-85a2-1debe9a16a65/volumes"
Sep 10 17:33:49.114850 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:49.114808    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ddde51-c250-49d2-a931-b43a53956de2" path="/var/lib/kubelet/pods/68ddde51-c250-49d2-a931-b43a53956de2/volumes"
Sep 10 17:33:49.712797 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:49.712767    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:33:56.360861 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.360830    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"]
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361148    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14955e52-c8a8-40b7-91e0-55421e9fa3a6" containerName="main"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361160    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="14955e52-c8a8-40b7-91e0-55421e9fa3a6" containerName="main"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361172    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81895ac3-7b87-4ac8-af51-cb9af0d6291e" containerName="main"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361178    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="81895ac3-7b87-4ac8-af51-cb9af0d6291e" containerName="main"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361185    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361191    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa87a685-95c2-4a27-aaf3-0ef4b07a5845" containerName="istio-proxy"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361196    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4829976d-6268-4603-9765-793343dead31" containerName="main"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361201    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4829976d-6268-4603-9765-793343dead31" containerName="main"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361207    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca6d684-11f1-4295-bebe-f12b9c3e5882" containerName="payload-pre-processing"
Sep 10 17:33:56.361218 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361211    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca6d684-11f1-4295-bebe-f12b9c3e5882" containerName="payload-pre-processing"
Sep 10 17:33:56.361520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361231    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ddde51-c250-49d2-a931-b43a53956de2" containerName="maas-api"
Sep 10 17:33:56.361520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361236    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ddde51-c250-49d2-a931-b43a53956de2" containerName="maas-api"
Sep 10 17:33:56.361520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361285    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca6d684-11f1-4295-bebe-f12b9c3e5882" containerName="payload-pre-processing"
Sep 10 17:33:56.361520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361294    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="81895ac3-7b87-4ac8-af51-cb9af0d6291e" containerName="main"
Sep 10 17:33:56.361520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361299    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="14955e52-c8a8-40b7-91e0-55421e9fa3a6" containerName="main"
Sep 10 17:33:56.361520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361305    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4829976d-6268-4603-9765-793343dead31" containerName="main"
Sep 10 17:33:56.361520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.361312    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ddde51-c250-49d2-a931-b43a53956de2" containerName="maas-api"
Sep 10 17:33:56.364231 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.364214    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.367761 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.367739    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5912a3-openshift-default-dockercfg-h88rv\""
Sep 10 17:33:56.367904 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.367887    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 17:33:56.381009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.380986    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"]
Sep 10 17:33:56.481571 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481518    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563b6a7a-2c9e-4a54-9e43-17854336886f-istiod-ca-cert\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481578    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bngk8\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-kube-api-access-bngk8\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481622    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-socket\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481647    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-data\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481708    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-token\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481752 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481750    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-credential-socket\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481964 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481790    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-certs\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481964 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481809    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-envoy\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.481964 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.481830    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-podinfo\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583267 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583232    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bngk8\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-kube-api-access-bngk8\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583289    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-socket\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-data\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583329    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-token\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583352    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-credential-socket\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583375    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-certs\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-envoy\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583414    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-podinfo\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583843 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583470    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563b6a7a-2c9e-4a54-9e43-17854336886f-istiod-ca-cert\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583843 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583795    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-data\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583954 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583835    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-certs\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.583954 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583927    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-credential-socket\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.584046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.583968    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-socket\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.584117 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.584100    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563b6a7a-2c9e-4a54-9e43-17854336886f-istiod-ca-cert\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.585794 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.585777    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-envoy\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.585926 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.585901    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-podinfo\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.592309 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.592280    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-token\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.592561 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.592518    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bngk8\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-kube-api-access-bngk8\") pod \"e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") " pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.673593 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.673503    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:56.952672 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.952648    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"]
Sep 10 17:33:56.954540 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:33:56.954491    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563b6a7a_2c9e_4a54_9e43_17854336886f.slice/crio-65bfe1ba8c00c0934117b1907561c67bdb50581bc52fc5c9497ae5b8102c79fd WatchSource:0}: Error finding container 65bfe1ba8c00c0934117b1907561c67bdb50581bc52fc5c9497ae5b8102c79fd: Status 404 returned error can't find the container with id 65bfe1ba8c00c0934117b1907561c67bdb50581bc52fc5c9497ae5b8102c79fd
Sep 10 17:33:56.956802 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.956768    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:33:56.956866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.956841    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:33:56.956910 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:56.956873    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:33:57.737080 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:57.737044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" event={"ID":"563b6a7a-2c9e-4a54-9e43-17854336886f","Type":"ContainerStarted","Data":"2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878"}
Sep 10 17:33:57.737446 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:57.737102    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" event={"ID":"563b6a7a-2c9e-4a54-9e43-17854336886f","Type":"ContainerStarted","Data":"65bfe1ba8c00c0934117b1907561c67bdb50581bc52fc5c9497ae5b8102c79fd"}
Sep 10 17:33:57.762071 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:57.762020    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podStartSLOduration=1.7620056659999999 podStartE2EDuration="1.762005666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:57.760121317 +0000 UTC m=+1233.116611718" watchObservedRunningTime="2026-09-10 17:33:57.762005666 +0000 UTC m=+1233.118496065"
Sep 10 17:33:58.674548 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:58.674498    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:58.679462 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:58.679437    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:58.743077 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:58.743053    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:33:58.744013 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:33:58.743992    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:34:04.270343 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.270308    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"]
Sep 10 17:34:04.273592 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.273576    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.278252 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.278229    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3c30a2-dockercfg-xvhhr\""
Sep 10 17:34:04.278365 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.278263    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3c30a2\""
Sep 10 17:34:04.305535 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.305503    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"]
Sep 10 17:34:04.356601 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.356569    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526pk\" (UniqueName: \"kubernetes.io/projected/7c9acb33-d0d2-407d-8fff-cf20e943fee2-kube-api-access-526pk\") pod \"payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.356778 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.356633    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9acb33-d0d2-407d-8fff-cf20e943fee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.360260 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.360234    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"]
Sep 10 17:34:04.363495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.363477    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.380483 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.380461    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"]
Sep 10 17:34:04.458155 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.458117    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9acb33-d0d2-407d-8fff-cf20e943fee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.458155 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.458160    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cs9j\" (UniqueName: \"kubernetes.io/projected/42fb6b85-280c-44c3-9391-298d3eb255f0-kube-api-access-4cs9j\") pod \"payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.458351 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.458219    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fb6b85-280c-44c3-9391-298d3eb255f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.458351 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.458239    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526pk\" (UniqueName: \"kubernetes.io/projected/7c9acb33-d0d2-407d-8fff-cf20e943fee2-kube-api-access-526pk\") pod \"payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.458839 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.458819    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9acb33-d0d2-407d-8fff-cf20e943fee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.472452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.472424    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-526pk\" (UniqueName: \"kubernetes.io/projected/7c9acb33-d0d2-407d-8fff-cf20e943fee2-kube-api-access-526pk\") pod \"payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.559072 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.559042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fb6b85-280c-44c3-9391-298d3eb255f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.559231 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.559122    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cs9j\" (UniqueName: \"kubernetes.io/projected/42fb6b85-280c-44c3-9391-298d3eb255f0-kube-api-access-4cs9j\") pod \"payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.559729 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.559709    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fb6b85-280c-44c3-9391-298d3eb255f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.573086 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.573059    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cs9j\" (UniqueName: \"kubernetes.io/projected/42fb6b85-280c-44c3-9391-298d3eb255f0-kube-api-access-4cs9j\") pod \"payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.583444 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.583424    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:04.675103 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.675063    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:04.871886 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.871791    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"]
Sep 10 17:34:04.876560 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:34:04.876519    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9acb33_d0d2_407d_8fff_cf20e943fee2.slice/crio-6a02360474543d5cf6434a488f38668cd677d88b7e0db9d349ba7347d958ff99 WatchSource:0}: Error finding container 6a02360474543d5cf6434a488f38668cd677d88b7e0db9d349ba7347d958ff99: Status 404 returned error can't find the container with id 6a02360474543d5cf6434a488f38668cd677d88b7e0db9d349ba7347d958ff99
Sep 10 17:34:04.978388 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:04.978341    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"]
Sep 10 17:34:04.980562 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:34:04.980518    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fb6b85_280c_44c3_9391_298d3eb255f0.slice/crio-ca88a205c522f8e0be62c2f7c72c3986103fe9983e7da43428ea71acaba21e5e WatchSource:0}: Error finding container ca88a205c522f8e0be62c2f7c72c3986103fe9983e7da43428ea71acaba21e5e: Status 404 returned error can't find the container with id ca88a205c522f8e0be62c2f7c72c3986103fe9983e7da43428ea71acaba21e5e
Sep 10 17:34:05.211424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.211324    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:34:05.778979 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.778936    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq" event={"ID":"42fb6b85-280c-44c3-9391-298d3eb255f0","Type":"ContainerStarted","Data":"ebae9337c6dc6dd8d273a8e60e4dcbedca34a6aafe2f31b2b31a975d1dfbd5ed"}
Sep 10 17:34:05.779454 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.779004    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq" event={"ID":"42fb6b85-280c-44c3-9391-298d3eb255f0","Type":"ContainerStarted","Data":"ca88a205c522f8e0be62c2f7c72c3986103fe9983e7da43428ea71acaba21e5e"}
Sep 10 17:34:05.779454 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.779174    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:05.781193 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.781162    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss" event={"ID":"7c9acb33-d0d2-407d-8fff-cf20e943fee2","Type":"ContainerStarted","Data":"20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709"}
Sep 10 17:34:05.781302 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.781213    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss" event={"ID":"7c9acb33-d0d2-407d-8fff-cf20e943fee2","Type":"ContainerStarted","Data":"6a02360474543d5cf6434a488f38668cd677d88b7e0db9d349ba7347d958ff99"}
Sep 10 17:34:05.781302 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.781292    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:05.812056 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.811989    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq" podStartSLOduration=1.811969363 podStartE2EDuration="1.811969363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:05.808676668 +0000 UTC m=+1241.165167084" watchObservedRunningTime="2026-09-10 17:34:05.811969363 +0000 UTC m=+1241.168459764"
Sep 10 17:34:05.839257 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:05.839194    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss" podStartSLOduration=1.8391737959999999 podStartE2EDuration="1.839173796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:05.836220758 +0000 UTC m=+1241.192711158" watchObservedRunningTime="2026-09-10 17:34:05.839173796 +0000 UTC m=+1241.195664197"
Sep 10 17:34:14.784699 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:14.784655    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 17:34:14.785082 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:14.784728    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 17:34:15.171737 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:15.171695    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:34:15.834600 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:15.834563    2554 generic.go:356] "Generic (PLEG): container finished" podID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerID="2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878" exitCode=0
Sep 10 17:34:15.835065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:15.834640    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" event={"ID":"563b6a7a-2c9e-4a54-9e43-17854336886f","Type":"ContainerDied","Data":"2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878"}
Sep 10 17:34:15.835148 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:15.835071    2554 scope.go:117] "RemoveContainer" containerID="2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878"
Sep 10 17:34:15.836132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:15.836100    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:15.836226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:15.836169    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:15.836226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:15.836204    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:16.788653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:16.788619    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:34:16.788837 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:16.788688    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:34:16.843738 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:16.843681    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" event={"ID":"563b6a7a-2c9e-4a54-9e43-17854336886f","Type":"ContainerStarted","Data":"15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8"}
Sep 10 17:34:17.674223 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:17.674125    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:34:17.674409 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:17.674378    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:34:17.674491 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:17.674435    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 17:34:18.674493 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:18.674451    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:34:18.675004 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:18.674511    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 17:34:19.674702 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:19.674663    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:34:19.675088 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:19.674716    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 17:34:20.674012 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:20.673976    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:34:20.674192 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:20.674033    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 17:34:21.674393 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:21.674354    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 10 17:34:21.674776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:21.674428    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 10 17:34:22.678363 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:22.678333    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:34:22.678774 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:22.678625    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:34:22.679329 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:22.679312    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:34:25.176412 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:25.176384    2554 scope.go:117] "RemoveContainer" containerID="5b82c34e8b96bee1be852977b7e574d5c6cb0dba609ae33edc36403d7ce96f40"
Sep 10 17:34:29.877709 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.877678    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"]
Sep 10 17:34:29.882541 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.882510    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.885161 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.885139    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c1fb61-openshift-default-dockercfg-tj6xc\""
Sep 10 17:34:29.902641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.902615    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"]
Sep 10 17:34:29.909137 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909114    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-certs\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909229 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909153    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863398a8-9b58-4fef-a919-d1b98e397783-istio-podinfo\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909229 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909175    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-data\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909229    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-credential-socket\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-socket\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909315 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863398a8-9b58-4fef-a919-d1b98e397783-istiod-ca-cert\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909329    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-istio-token\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909396    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-envoy\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:29.909456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:29.909441    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44d7\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-kube-api-access-d44d7\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010509 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010473    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-socket\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010509 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010508    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863398a8-9b58-4fef-a919-d1b98e397783-istiod-ca-cert\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010587    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-istio-token\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010607    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-envoy\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010626    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44d7\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-kube-api-access-d44d7\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010678    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-certs\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010707    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863398a8-9b58-4fef-a919-d1b98e397783-istio-podinfo\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-data\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.010792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010780    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-credential-socket\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.011152 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.010973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-socket\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.011152 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.011129    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-certs\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.011221 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.011198    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-data\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.011270 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.011251    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-credential-socket\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.011383 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.011367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863398a8-9b58-4fef-a919-d1b98e397783-istiod-ca-cert\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.013136 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.013114    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-envoy\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.013280 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.013263    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863398a8-9b58-4fef-a919-d1b98e397783-istio-podinfo\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.026806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.026779    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-istio-token\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.030329 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.030306    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44d7\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-kube-api-access-d44d7\") pod \"e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") " pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.192883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.192784    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:30.482230 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.482197    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"]
Sep 10 17:34:30.484441 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:34:30.484412    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863398a8_9b58_4fef_a919_d1b98e397783.slice/crio-f17a70072b95ad1ab7e25a9722a044add80a9dc3ab502aa099ae41653a46b30b WatchSource:0}: Error finding container f17a70072b95ad1ab7e25a9722a044add80a9dc3ab502aa099ae41653a46b30b: Status 404 returned error can't find the container with id f17a70072b95ad1ab7e25a9722a044add80a9dc3ab502aa099ae41653a46b30b
Sep 10 17:34:30.486729 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.486698    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:30.486798 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.486759    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:30.486798 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.486790    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:30.925406 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.925367    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" event={"ID":"863398a8-9b58-4fef-a919-d1b98e397783","Type":"ContainerStarted","Data":"1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb"}
Sep 10 17:34:30.925824 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.925430    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" event={"ID":"863398a8-9b58-4fef-a919-d1b98e397783","Type":"ContainerStarted","Data":"f17a70072b95ad1ab7e25a9722a044add80a9dc3ab502aa099ae41653a46b30b"}
Sep 10 17:34:30.957408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:30.957361    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podStartSLOduration=1.957346422 podStartE2EDuration="1.957346422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:30.954130243 +0000 UTC m=+1266.310620643" watchObservedRunningTime="2026-09-10 17:34:30.957346422 +0000 UTC m=+1266.313836822"
Sep 10 17:34:31.193797 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:31.193702    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:31.198820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:31.198794    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:31.933692 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:31.933647    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:31.937067 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:31.937029    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:38.012582 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.012545    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"]
Sep 10 17:34:38.017028 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.017011    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.019907 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.019882    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c1fb61\""
Sep 10 17:34:38.020013 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.019968    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c1fb61-dockercfg-jpzvs\""
Sep 10 17:34:38.036986 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.036964    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"]
Sep 10 17:34:38.091006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.090981    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpcs\" (UniqueName: \"kubernetes.io/projected/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-kube-api-access-rlpcs\") pod \"payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.091152 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.091028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.192181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.192150    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpcs\" (UniqueName: \"kubernetes.io/projected/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-kube-api-access-rlpcs\") pod \"payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.192354 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.192189    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.192983 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.192961    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.201484 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.201460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpcs\" (UniqueName: \"kubernetes.io/projected/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-kube-api-access-rlpcs\") pod \"payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.327027 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.326997    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:38.615096 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.615066    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"]
Sep 10 17:34:38.616169 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:34:38.616143    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56cb3fe_3eaa_4340_9a51_3cff321c98b1.slice/crio-b4d2ba5d9b03fcd4bcd618016c039d682cf9c354f44141f7d330ea7429d368b6 WatchSource:0}: Error finding container b4d2ba5d9b03fcd4bcd618016c039d682cf9c354f44141f7d330ea7429d368b6: Status 404 returned error can't find the container with id b4d2ba5d9b03fcd4bcd618016c039d682cf9c354f44141f7d330ea7429d368b6
Sep 10 17:34:38.969980 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.969885    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7" event={"ID":"c56cb3fe-3eaa-4340-9a51-3cff321c98b1","Type":"ContainerStarted","Data":"f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb"}
Sep 10 17:34:38.969980 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.969951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7" event={"ID":"c56cb3fe-3eaa-4340-9a51-3cff321c98b1","Type":"ContainerStarted","Data":"b4d2ba5d9b03fcd4bcd618016c039d682cf9c354f44141f7d330ea7429d368b6"}
Sep 10 17:34:38.970177 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:38.969995    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:39.031068 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:39.031003    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7" podStartSLOduration=2.030981721 podStartE2EDuration="2.030981721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:39.029221209 +0000 UTC m=+1274.385711608" watchObservedRunningTime="2026-09-10 17:34:39.030981721 +0000 UTC m=+1274.387472122"
Sep 10 17:34:41.483028 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:41.482994    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:34:44.280693 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.280656    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"]
Sep 10 17:34:44.284297 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.284275    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.288051 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.288030    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-68cae2e9\""
Sep 10 17:34:44.297071 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.297049    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"]
Sep 10 17:34:44.370072 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.370037    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwn2\" (UniqueName: \"kubernetes.io/projected/175c8fe5-f021-421a-9a01-a258ec4a6bb7-kube-api-access-xrwn2\") pod \"payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.370278 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.370109    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c8fe5-f021-421a-9a01-a258ec4a6bb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.471053 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.471012    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwn2\" (UniqueName: \"kubernetes.io/projected/175c8fe5-f021-421a-9a01-a258ec4a6bb7-kube-api-access-xrwn2\") pod \"payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.471240 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.471123    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c8fe5-f021-421a-9a01-a258ec4a6bb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.471875 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.471852    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c8fe5-f021-421a-9a01-a258ec4a6bb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.480045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.480018    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwn2\" (UniqueName: \"kubernetes.io/projected/175c8fe5-f021-421a-9a01-a258ec4a6bb7-kube-api-access-xrwn2\") pod \"payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.595102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.595064    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:44.895770 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:44.895734    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"]
Sep 10 17:34:44.897171 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:34:44.897141    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175c8fe5_f021_421a_9a01_a258ec4a6bb7.slice/crio-36eefe02aa66c946beebc0ab61ed9bcd3dfcc727480932c7b3a42a1ff6c9c8d5 WatchSource:0}: Error finding container 36eefe02aa66c946beebc0ab61ed9bcd3dfcc727480932c7b3a42a1ff6c9c8d5: Status 404 returned error can't find the container with id 36eefe02aa66c946beebc0ab61ed9bcd3dfcc727480932c7b3a42a1ff6c9c8d5
Sep 10 17:34:45.003515 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:45.003486    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj" event={"ID":"175c8fe5-f021-421a-9a01-a258ec4a6bb7","Type":"ContainerStarted","Data":"bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155"}
Sep 10 17:34:45.003648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:45.003560    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj" event={"ID":"175c8fe5-f021-421a-9a01-a258ec4a6bb7","Type":"ContainerStarted","Data":"36eefe02aa66c946beebc0ab61ed9bcd3dfcc727480932c7b3a42a1ff6c9c8d5"}
Sep 10 17:34:45.003713 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:45.003696    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:45.033800 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:45.033751    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj" podStartSLOduration=1.033735793 podStartE2EDuration="1.033735793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:45.031398908 +0000 UTC m=+1280.387889307" watchObservedRunningTime="2026-09-10 17:34:45.033735793 +0000 UTC m=+1280.390226193"
Sep 10 17:34:49.976363 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:49.976330    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:34:52.053804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:52.053766    2554 generic.go:356] "Generic (PLEG): container finished" podID="863398a8-9b58-4fef-a919-d1b98e397783" containerID="1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb" exitCode=0
Sep 10 17:34:52.054207 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:52.053837    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" event={"ID":"863398a8-9b58-4fef-a919-d1b98e397783","Type":"ContainerDied","Data":"1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb"}
Sep 10 17:34:52.054207 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:52.054192    2554 scope.go:117] "RemoveContainer" containerID="1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb"
Sep 10 17:34:52.055360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:52.055330    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:52.055417 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:52.055403    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:52.055457 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:52.055434    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 17:34:53.060698 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:53.060658    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" event={"ID":"863398a8-9b58-4fef-a919-d1b98e397783","Type":"ContainerStarted","Data":"210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d"}
Sep 10 17:34:53.193584 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:53.193553    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:34:53.193749 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:53.193730    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:34:53.193795 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:53.193775    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 17:34:54.080677 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:54.080641    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:34:54.193652 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:54.193617    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:34:54.193814 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:54.193667    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 17:34:55.193337 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:55.193294    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:34:55.193856 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:55.193367    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 17:34:56.011724 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:56.011687    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:34:56.193880 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:56.193844    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:34:56.194264 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:56.193910    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 17:34:57.193823 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:57.193792    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:34:57.194001 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:57.193853    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 17:34:58.193920 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:58.193885    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:34:58.194111 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:58.193936    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 17:34:59.193650 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:59.193605    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 17:34:59.193858 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:34:59.193688    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 17:35:00.197814 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:00.197778    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:35:00.198180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:00.198063    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:35:00.198810 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:00.198792    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:35:11.011236 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.011201    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"]
Sep 10 17:35:11.014792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.014775    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.017566 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.017523    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"auth-test-model-3c30a2-kserve-self-signed-certs\""
Sep 10 17:35:11.018640 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.018623    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"default-dockercfg-r5g7h\""
Sep 10 17:35:11.018699 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.018674    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"kube-root-ca.crt\""
Sep 10 17:35:11.018747 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.018625    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"openshift-service-ca.crt\""
Sep 10 17:35:11.024452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.024427    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"]
Sep 10 17:35:11.125769 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.125740    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/ccaeba79-8f73-4867-90d4-440328f2494a-kube-api-access-mwmq7\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.125933 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.125777    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccaeba79-8f73-4867-90d4-440328f2494a-tls-certs\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.125933 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.125812    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-model-cache\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.125933 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.125868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-tmp-dir\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.126047 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.125962    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-dshm\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.126096 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.126081    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-home\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.227494 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227461    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-home\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.227686 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/ccaeba79-8f73-4867-90d4-440328f2494a-kube-api-access-mwmq7\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.227686 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccaeba79-8f73-4867-90d4-440328f2494a-tls-certs\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.227807 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227692    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-model-cache\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.227807 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227714    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-tmp-dir\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.227807 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227774    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-dshm\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.228000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227836    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-home\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.228062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.227995    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-model-cache\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.228115 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.228082    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-tmp-dir\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.229974 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.229947    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-dshm\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.230289 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.230266    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccaeba79-8f73-4867-90d4-440328f2494a-tls-certs\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.237872 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.237851    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/ccaeba79-8f73-4867-90d4-440328f2494a-kube-api-access-mwmq7\") pod \"auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.324889 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.324858    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:11.608093 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:11.608067    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"]
Sep 10 17:35:11.610599 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:35:11.610566    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccaeba79_8f73_4867_90d4_440328f2494a.slice/crio-99c09ec450804e65e9817b9cec4903649d944d74f9eb235a5b57bb65f30882c3 WatchSource:0}: Error finding container 99c09ec450804e65e9817b9cec4903649d944d74f9eb235a5b57bb65f30882c3: Status 404 returned error can't find the container with id 99c09ec450804e65e9817b9cec4903649d944d74f9eb235a5b57bb65f30882c3
Sep 10 17:35:12.159164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:12.159117    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv" event={"ID":"ccaeba79-8f73-4867-90d4-440328f2494a","Type":"ContainerStarted","Data":"6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a"}
Sep 10 17:35:12.159164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:12.159169    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv" event={"ID":"ccaeba79-8f73-4867-90d4-440328f2494a","Type":"ContainerStarted","Data":"99c09ec450804e65e9817b9cec4903649d944d74f9eb235a5b57bb65f30882c3"}
Sep 10 17:35:12.205161 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:12.205116    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv" podStartSLOduration=2.205100599 podStartE2EDuration="2.205100599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:12.204291837 +0000 UTC m=+1307.560782238" watchObservedRunningTime="2026-09-10 17:35:12.205100599 +0000 UTC m=+1307.561591017"
Sep 10 17:35:15.597000 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:15.596961    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:35:21.325792 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:21.325739    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:21.326236 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:21.325813    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:21.338482 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:21.338458    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:22.219150 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:22.219120    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:35:25.208510 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:25.208479    2554 scope.go:117] "RemoveContainer" containerID="419177e01eac6ba827d75485057671c90f97fff94a30eeb0a9f04c77340aed3e"
Sep 10 17:35:33.759338 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.759304    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"]
Sep 10 17:35:33.764154 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.764136    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.768241 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.768215    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-68cae2e9\"/\"default-dockercfg-q6pgf\""
Sep 10 17:35:33.768610 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.768589    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-68cae2e9\"/\"kube-root-ca.crt\""
Sep 10 17:35:33.769377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.769356    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-68cae2e9\"/\"openshift-service-ca.crt\""
Sep 10 17:35:33.769459 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.769443    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-68cae2e9\"/\"e2e-lifecycle-model-68cae2e9-kserve-self-signed-certs\""
Sep 10 17:35:33.776786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.776765    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"]
Sep 10 17:35:33.833840 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.833797    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99v7\" (UniqueName: \"kubernetes.io/projected/c865b57e-6eed-41ac-8e18-b193c62f5555-kube-api-access-b99v7\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.834016 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.833864    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-dshm\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.834016 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.833967    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c865b57e-6eed-41ac-8e18-b193c62f5555-tls-certs\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.834016 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.834007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-tmp-dir\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.834132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.834053    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-model-cache\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.834132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.834080    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-home\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935245 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935200    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99v7\" (UniqueName: \"kubernetes.io/projected/c865b57e-6eed-41ac-8e18-b193c62f5555-kube-api-access-b99v7\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935449 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935284    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-dshm\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935449 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935361    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c865b57e-6eed-41ac-8e18-b193c62f5555-tls-certs\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935449 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-tmp-dir\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935449 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-model-cache\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935705 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935477    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-home\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935890 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935865    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-model-cache\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935971 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935885    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-home\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.935971 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.935913    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-tmp-dir\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.937657 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.937630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-dshm\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.937979 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.937958    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c865b57e-6eed-41ac-8e18-b193c62f5555-tls-certs\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:33.943810 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:33.943785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99v7\" (UniqueName: \"kubernetes.io/projected/c865b57e-6eed-41ac-8e18-b193c62f5555-kube-api-access-b99v7\") pod \"e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") " pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:34.073949 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:34.073911    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:34.352503 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:34.352475    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"]
Sep 10 17:35:34.355284 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:35:34.355242    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc865b57e_6eed_41ac_8e18_b193c62f5555.slice/crio-9424c001fb724abdd6100ab44483a0fb8395e8b8640a7e5a723e81dc7619f2f9 WatchSource:0}: Error finding container 9424c001fb724abdd6100ab44483a0fb8395e8b8640a7e5a723e81dc7619f2f9: Status 404 returned error can't find the container with id 9424c001fb724abdd6100ab44483a0fb8395e8b8640a7e5a723e81dc7619f2f9
Sep 10 17:35:35.274997 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:35.274955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts" event={"ID":"c865b57e-6eed-41ac-8e18-b193c62f5555","Type":"ContainerStarted","Data":"547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f"}
Sep 10 17:35:35.274997 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:35.275004    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts" event={"ID":"c865b57e-6eed-41ac-8e18-b193c62f5555","Type":"ContainerStarted","Data":"9424c001fb724abdd6100ab44483a0fb8395e8b8640a7e5a723e81dc7619f2f9"}
Sep 10 17:35:35.294835 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:35.294790    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts" podStartSLOduration=2.294776067 podStartE2EDuration="2.294776067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:35.293145689 +0000 UTC m=+1330.649636096" watchObservedRunningTime="2026-09-10 17:35:35.294776067 +0000 UTC m=+1330.651266467"
Sep 10 17:35:38.695942 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:38.695905    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:35:44.074339 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:44.074299    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:44.074846 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:44.074364    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:44.087290 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:44.087264    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:44.331714 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:44.331624    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:35:59.683596 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:35:59.683499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:36:03.144422 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.144340    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"]
Sep 10 17:36:03.148249 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.148232    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"
Sep 10 17:36:03.151710 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.151689    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:36:03.152673 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.152653    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:36:03.162846 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.162824    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"]
Sep 10 17:36:03.223181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.223150    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb960c9-4721-4e85-bd78-827da85a109c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf\" (UID: \"4bb960c9-4721-4e85-bd78-827da85a109c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"
Sep 10 17:36:03.324739 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.324687    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb960c9-4721-4e85-bd78-827da85a109c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf\" (UID: \"4bb960c9-4721-4e85-bd78-827da85a109c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"
Sep 10 17:36:03.325328 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.325308    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb960c9-4721-4e85-bd78-827da85a109c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf\" (UID: \"4bb960c9-4721-4e85-bd78-827da85a109c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"
Sep 10 17:36:03.458445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.458371    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"
Sep 10 17:36:03.745360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:03.745335    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"]
Sep 10 17:36:03.747517 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:36:03.747488    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb960c9_4721_4e85_bd78_827da85a109c.slice/crio-b28db4ad9912fd3114b33cb87e9759e6aea8cb8919a3ff4a79479dbaa2de2db8 WatchSource:0}: Error finding container b28db4ad9912fd3114b33cb87e9759e6aea8cb8919a3ff4a79479dbaa2de2db8: Status 404 returned error can't find the container with id b28db4ad9912fd3114b33cb87e9759e6aea8cb8919a3ff4a79479dbaa2de2db8
Sep 10 17:36:04.419543 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:04.419500    2554 generic.go:356] "Generic (PLEG): container finished" podID="4bb960c9-4721-4e85-bd78-827da85a109c" containerID="6261c99e801ebdb4dee0b29f615a55a90abdb44a98b3dd71dcc26e975e72b8ee" exitCode=0
Sep 10 17:36:04.419901 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:04.419591    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf" event={"ID":"4bb960c9-4721-4e85-bd78-827da85a109c","Type":"ContainerDied","Data":"6261c99e801ebdb4dee0b29f615a55a90abdb44a98b3dd71dcc26e975e72b8ee"}
Sep 10 17:36:04.419901 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:04.419678    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf" event={"ID":"4bb960c9-4721-4e85-bd78-827da85a109c","Type":"ContainerStarted","Data":"b28db4ad9912fd3114b33cb87e9759e6aea8cb8919a3ff4a79479dbaa2de2db8"}
Sep 10 17:36:05.558144 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:05.558120    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"
Sep 10 17:36:05.644261 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:05.644230    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb960c9-4721-4e85-bd78-827da85a109c-maas-api-service-ca\") pod \"4bb960c9-4721-4e85-bd78-827da85a109c\" (UID: \"4bb960c9-4721-4e85-bd78-827da85a109c\") "
Sep 10 17:36:05.644635 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:05.644613    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb960c9-4721-4e85-bd78-827da85a109c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4bb960c9-4721-4e85-bd78-827da85a109c" (UID: "4bb960c9-4721-4e85-bd78-827da85a109c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:36:05.745734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:05.745657    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb960c9-4721-4e85-bd78-827da85a109c-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:06.430656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:06.430630    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"
Sep 10 17:36:06.430656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:06.430651    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf" event={"ID":"4bb960c9-4721-4e85-bd78-827da85a109c","Type":"ContainerDied","Data":"b28db4ad9912fd3114b33cb87e9759e6aea8cb8919a3ff4a79479dbaa2de2db8"}
Sep 10 17:36:06.430866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:06.430693    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28db4ad9912fd3114b33cb87e9759e6aea8cb8919a3ff4a79479dbaa2de2db8"
Sep 10 17:36:18.896249 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:18.896212    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"]
Sep 10 17:36:18.896896 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:18.896422    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj" podUID="175c8fe5-f021-421a-9a01-a258ec4a6bb7" containerName="payload-pre-processing" containerID="cri-o://bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155" gracePeriod=30
Sep 10 17:36:19.115497 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.115468    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"]
Sep 10 17:36:19.115667 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.115507    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68cae2e9-m8pnf"]
Sep 10 17:36:19.151420 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.151365    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:36:19.272955 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.272919    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrwn2\" (UniqueName: \"kubernetes.io/projected/175c8fe5-f021-421a-9a01-a258ec4a6bb7-kube-api-access-xrwn2\") pod \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") "
Sep 10 17:36:19.273120 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.272983    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c8fe5-f021-421a-9a01-a258ec4a6bb7-plugins-config-volume\") pod \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\" (UID: \"175c8fe5-f021-421a-9a01-a258ec4a6bb7\") "
Sep 10 17:36:19.273414 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.273379    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175c8fe5-f021-421a-9a01-a258ec4a6bb7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "175c8fe5-f021-421a-9a01-a258ec4a6bb7" (UID: "175c8fe5-f021-421a-9a01-a258ec4a6bb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:36:19.275181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.275161    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175c8fe5-f021-421a-9a01-a258ec4a6bb7-kube-api-access-xrwn2" (OuterVolumeSpecName: "kube-api-access-xrwn2") pod "175c8fe5-f021-421a-9a01-a258ec4a6bb7" (UID: "175c8fe5-f021-421a-9a01-a258ec4a6bb7"). InnerVolumeSpecName "kube-api-access-xrwn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:36:19.373905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.373873    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwn2\" (UniqueName: \"kubernetes.io/projected/175c8fe5-f021-421a-9a01-a258ec4a6bb7-kube-api-access-xrwn2\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:19.373905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.373900    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c8fe5-f021-421a-9a01-a258ec4a6bb7-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:19.507162 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.507067    2554 generic.go:356] "Generic (PLEG): container finished" podID="175c8fe5-f021-421a-9a01-a258ec4a6bb7" containerID="bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155" exitCode=0
Sep 10 17:36:19.507162 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.507110    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"
Sep 10 17:36:19.507162 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.507122    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj" event={"ID":"175c8fe5-f021-421a-9a01-a258ec4a6bb7","Type":"ContainerDied","Data":"bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155"}
Sep 10 17:36:19.507427 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.507166    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj" event={"ID":"175c8fe5-f021-421a-9a01-a258ec4a6bb7","Type":"ContainerDied","Data":"36eefe02aa66c946beebc0ab61ed9bcd3dfcc727480932c7b3a42a1ff6c9c8d5"}
Sep 10 17:36:19.507427 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.507188    2554 scope.go:117] "RemoveContainer" containerID="bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155"
Sep 10 17:36:19.516411 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.516391    2554 scope.go:117] "RemoveContainer" containerID="bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155"
Sep 10 17:36:19.516719 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:36:19.516702    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155\": container with ID starting with bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155 not found: ID does not exist" containerID="bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155"
Sep 10 17:36:19.516779 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.516728    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155"} err="failed to get container status \"bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155\": rpc error: code = NotFound desc = could not find container \"bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155\": container with ID starting with bbf357656a98a039f7ec5ee949d927a4194b62297bad42b60825dcd31fa6a155 not found: ID does not exist"
Sep 10 17:36:19.529726 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.529703    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"]
Sep 10 17:36:19.537668 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:19.537641    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-68cae2e9-647b965d5d-w7nkj"]
Sep 10 17:36:20.081006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:20.080972    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:36:21.114997 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:21.114967    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175c8fe5-f021-421a-9a01-a258ec4a6bb7" path="/var/lib/kubelet/pods/175c8fe5-f021-421a-9a01-a258ec4a6bb7/volumes"
Sep 10 17:36:21.115369 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:21.115293    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb960c9-4721-4e85-bd78-827da85a109c" path="/var/lib/kubelet/pods/4bb960c9-4721-4e85-bd78-827da85a109c/volumes"
Sep 10 17:36:25.228438 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:25.228411    2554 scope.go:117] "RemoveContainer" containerID="eac25a0258617659ba9cb3023e4cd4bbe198296a06f8f07ec770f6e34d493e9a"
Sep 10 17:36:25.236780 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:25.236756    2554 scope.go:117] "RemoveContainer" containerID="3dd7d5e23cd34caa6ad542178afdd59acab04dd4ca952c43b6a6460255c16fad"
Sep 10 17:36:25.244441 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:25.244423    2554 scope.go:117] "RemoveContainer" containerID="85ee014cbc019c4bc119f4615b924a2d2cfd6df1fee4584a94c7c8b9faf982e5"
Sep 10 17:36:27.202541 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:27.202497    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"]
Sep 10 17:36:36.174100 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:36.174064    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:36:42.222230 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.222182    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts" podUID="c865b57e-6eed-41ac-8e18-b193c62f5555" containerName="main" containerID="cri-o://547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f" gracePeriod=45
Sep 10 17:36:42.475234 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.475176    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:36:42.596945 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.596914    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-tmp-dir\") pod \"c865b57e-6eed-41ac-8e18-b193c62f5555\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") "
Sep 10 17:36:42.596945 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.596949    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b99v7\" (UniqueName: \"kubernetes.io/projected/c865b57e-6eed-41ac-8e18-b193c62f5555-kube-api-access-b99v7\") pod \"c865b57e-6eed-41ac-8e18-b193c62f5555\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") "
Sep 10 17:36:42.597180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.596988    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-dshm\") pod \"c865b57e-6eed-41ac-8e18-b193c62f5555\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") "
Sep 10 17:36:42.597180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597021    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c865b57e-6eed-41ac-8e18-b193c62f5555-tls-certs\") pod \"c865b57e-6eed-41ac-8e18-b193c62f5555\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") "
Sep 10 17:36:42.597180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597069    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-model-cache\") pod \"c865b57e-6eed-41ac-8e18-b193c62f5555\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") "
Sep 10 17:36:42.597180 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597117    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-home\") pod \"c865b57e-6eed-41ac-8e18-b193c62f5555\" (UID: \"c865b57e-6eed-41ac-8e18-b193c62f5555\") "
Sep 10 17:36:42.597400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597218    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c865b57e-6eed-41ac-8e18-b193c62f5555" (UID: "c865b57e-6eed-41ac-8e18-b193c62f5555"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:36:42.597477 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597439    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-model-cache" (OuterVolumeSpecName: "model-cache") pod "c865b57e-6eed-41ac-8e18-b193c62f5555" (UID: "c865b57e-6eed-41ac-8e18-b193c62f5555"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:36:42.597477 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597455    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-home" (OuterVolumeSpecName: "home") pod "c865b57e-6eed-41ac-8e18-b193c62f5555" (UID: "c865b57e-6eed-41ac-8e18-b193c62f5555"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:36:42.597619 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597597    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:42.597661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597624    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:42.597661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.597640    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:42.599246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.599221    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-dshm" (OuterVolumeSpecName: "dshm") pod "c865b57e-6eed-41ac-8e18-b193c62f5555" (UID: "c865b57e-6eed-41ac-8e18-b193c62f5555"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:36:42.599352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.599244    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c865b57e-6eed-41ac-8e18-b193c62f5555-kube-api-access-b99v7" (OuterVolumeSpecName: "kube-api-access-b99v7") pod "c865b57e-6eed-41ac-8e18-b193c62f5555" (UID: "c865b57e-6eed-41ac-8e18-b193c62f5555"). InnerVolumeSpecName "kube-api-access-b99v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:36:42.599352 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.599224    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c865b57e-6eed-41ac-8e18-b193c62f5555-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c865b57e-6eed-41ac-8e18-b193c62f5555" (UID: "c865b57e-6eed-41ac-8e18-b193c62f5555"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:36:42.623364 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.623340    2554 generic.go:356] "Generic (PLEG): container finished" podID="c865b57e-6eed-41ac-8e18-b193c62f5555" containerID="547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f" exitCode=0
Sep 10 17:36:42.623466 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.623383    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"
Sep 10 17:36:42.623466 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.623416    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts" event={"ID":"c865b57e-6eed-41ac-8e18-b193c62f5555","Type":"ContainerDied","Data":"547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f"}
Sep 10 17:36:42.623466 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.623448    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts" event={"ID":"c865b57e-6eed-41ac-8e18-b193c62f5555","Type":"ContainerDied","Data":"9424c001fb724abdd6100ab44483a0fb8395e8b8640a7e5a723e81dc7619f2f9"}
Sep 10 17:36:42.623466 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.623465    2554 scope.go:117] "RemoveContainer" containerID="547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f"
Sep 10 17:36:42.631795 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.631777    2554 scope.go:117] "RemoveContainer" containerID="547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f"
Sep 10 17:36:42.632077 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:36:42.632056    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f\": container with ID starting with 547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f not found: ID does not exist" containerID="547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f"
Sep 10 17:36:42.632138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.632086    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f"} err="failed to get container status \"547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f\": rpc error: code = NotFound desc = could not find container \"547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f\": container with ID starting with 547a89f58b9e76aa0620ef9015d91aed044ff821585bca8b3062d9caa0967c3f not found: ID does not exist"
Sep 10 17:36:42.644670 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.644649    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"]
Sep 10 17:36:42.651139 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.651119    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-68cae2e9/e2e-lifecycle-model-68cae2e9-kserve-8469644c5-2hsts"]
Sep 10 17:36:42.698387 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.698360    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c865b57e-6eed-41ac-8e18-b193c62f5555-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:42.698387 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.698386    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c865b57e-6eed-41ac-8e18-b193c62f5555-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:42.698581 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:42.698398    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99v7\" (UniqueName: \"kubernetes.io/projected/c865b57e-6eed-41ac-8e18-b193c62f5555-kube-api-access-b99v7\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:36:43.114762 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:43.114726    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c865b57e-6eed-41ac-8e18-b193c62f5555" path="/var/lib/kubelet/pods/c865b57e-6eed-41ac-8e18-b193c62f5555/volumes"
Sep 10 17:36:52.568681 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:36:52.568644    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:37:09.471263 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:37:09.471225    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:37:25.272976 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:37:25.272889    2554 scope.go:117] "RemoveContainer" containerID="27b91b486b378f16d11968a3c39f0b859cc88e46930f839eb992365d0b564907"
Sep 10 17:37:25.865878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:37:25.865845    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:37:41.262694 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:37:41.262660    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:37:57.476114 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:37:57.476078    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:38:12.084656 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.084620    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:38:12.803023 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.802988    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"]
Sep 10 17:38:12.803355 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803343    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175c8fe5-f021-421a-9a01-a258ec4a6bb7" containerName="payload-pre-processing"
Sep 10 17:38:12.803401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803357    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="175c8fe5-f021-421a-9a01-a258ec4a6bb7" containerName="payload-pre-processing"
Sep 10 17:38:12.803401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803376    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb960c9-4721-4e85-bd78-827da85a109c" containerName="revoke-keys"
Sep 10 17:38:12.803401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803382    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb960c9-4721-4e85-bd78-827da85a109c" containerName="revoke-keys"
Sep 10 17:38:12.803401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803391    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c865b57e-6eed-41ac-8e18-b193c62f5555" containerName="main"
Sep 10 17:38:12.803401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803397    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c865b57e-6eed-41ac-8e18-b193c62f5555" containerName="main"
Sep 10 17:38:12.803594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803452    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb960c9-4721-4e85-bd78-827da85a109c" containerName="revoke-keys"
Sep 10 17:38:12.803594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803460    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="175c8fe5-f021-421a-9a01-a258ec4a6bb7" containerName="payload-pre-processing"
Sep 10 17:38:12.803594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.803470    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c865b57e-6eed-41ac-8e18-b193c62f5555" containerName="main"
Sep 10 17:38:12.806790 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.806773    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:12.809514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.809490    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b5f818-dockercfg-p9d4h\""
Sep 10 17:38:12.809665 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.809493    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b5f818\""
Sep 10 17:38:12.818630 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.818607    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"]
Sep 10 17:38:12.963371 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.963340    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2659b0-336e-4b43-9998-bb05f275bb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:12.963576 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:12.963458    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn4r\" (UniqueName: \"kubernetes.io/projected/5e2659b0-336e-4b43-9998-bb05f275bb09-kube-api-access-wxn4r\") pod \"payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:13.064776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:13.064680    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn4r\" (UniqueName: \"kubernetes.io/projected/5e2659b0-336e-4b43-9998-bb05f275bb09-kube-api-access-wxn4r\") pod \"payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:13.064776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:13.064736    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2659b0-336e-4b43-9998-bb05f275bb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:13.065373 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:13.065352    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2659b0-336e-4b43-9998-bb05f275bb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:13.079461 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:13.079427    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn4r\" (UniqueName: \"kubernetes.io/projected/5e2659b0-336e-4b43-9998-bb05f275bb09-kube-api-access-wxn4r\") pod \"payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:13.117933 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:13.117903    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:13.390906 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:13.390876    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"]
Sep 10 17:38:13.392368 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:38:13.392329    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2659b0_336e_4b43_9998_bb05f275bb09.slice/crio-95ea4dc7e769b8dc6c35d1299ea73f22f2a41302e1d83b7d0c2a33ac7261121a WatchSource:0}: Error finding container 95ea4dc7e769b8dc6c35d1299ea73f22f2a41302e1d83b7d0c2a33ac7261121a: Status 404 returned error can't find the container with id 95ea4dc7e769b8dc6c35d1299ea73f22f2a41302e1d83b7d0c2a33ac7261121a
Sep 10 17:38:14.080296 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:14.080257    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k" event={"ID":"5e2659b0-336e-4b43-9998-bb05f275bb09","Type":"ContainerStarted","Data":"815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528"}
Sep 10 17:38:14.080494 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:14.080338    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k" event={"ID":"5e2659b0-336e-4b43-9998-bb05f275bb09","Type":"ContainerStarted","Data":"95ea4dc7e769b8dc6c35d1299ea73f22f2a41302e1d83b7d0c2a33ac7261121a"}
Sep 10 17:38:14.080494 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:14.080359    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:14.100378 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:14.100331    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k" podStartSLOduration=2.100315188 podStartE2EDuration="2.100315188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:14.099982526 +0000 UTC m=+1489.456472927" watchObservedRunningTime="2026-09-10 17:38:14.100315188 +0000 UTC m=+1489.456805588"
Sep 10 17:38:18.635188 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:18.635152    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"]
Sep 10 17:38:25.087474 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:25.087441    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:38:25.115505 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:25.115473    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:38:25.118051 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:25.118023    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:38:30.273923 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:30.273889    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:38:33.654423 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.654361    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv" podUID="ccaeba79-8f73-4867-90d4-440328f2494a" containerName="main" containerID="cri-o://6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a" gracePeriod=45
Sep 10 17:38:33.912468 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.912405    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:38:33.964791 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.964760    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-model-cache\") pod \"ccaeba79-8f73-4867-90d4-440328f2494a\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") "
Sep 10 17:38:33.964978 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.964799    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccaeba79-8f73-4867-90d4-440328f2494a-tls-certs\") pod \"ccaeba79-8f73-4867-90d4-440328f2494a\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") "
Sep 10 17:38:33.964978 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.964899    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-tmp-dir\") pod \"ccaeba79-8f73-4867-90d4-440328f2494a\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") "
Sep 10 17:38:33.964978 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.964920    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/ccaeba79-8f73-4867-90d4-440328f2494a-kube-api-access-mwmq7\") pod \"ccaeba79-8f73-4867-90d4-440328f2494a\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") "
Sep 10 17:38:33.965164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965051    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ccaeba79-8f73-4867-90d4-440328f2494a" (UID: "ccaeba79-8f73-4867-90d4-440328f2494a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:33.965164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965089    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-dshm\") pod \"ccaeba79-8f73-4867-90d4-440328f2494a\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") "
Sep 10 17:38:33.965164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965126    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-home\") pod \"ccaeba79-8f73-4867-90d4-440328f2494a\" (UID: \"ccaeba79-8f73-4867-90d4-440328f2494a\") "
Sep 10 17:38:33.965399 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965179    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ccaeba79-8f73-4867-90d4-440328f2494a" (UID: "ccaeba79-8f73-4867-90d4-440328f2494a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:33.965456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965394    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-home" (OuterVolumeSpecName: "home") pod "ccaeba79-8f73-4867-90d4-440328f2494a" (UID: "ccaeba79-8f73-4867-90d4-440328f2494a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:33.965641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965620    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:38:33.965725 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965645    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:38:33.965725 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.965658    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:38:33.967282 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.967257    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccaeba79-8f73-4867-90d4-440328f2494a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ccaeba79-8f73-4867-90d4-440328f2494a" (UID: "ccaeba79-8f73-4867-90d4-440328f2494a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:38:33.967464 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.967446    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-dshm" (OuterVolumeSpecName: "dshm") pod "ccaeba79-8f73-4867-90d4-440328f2494a" (UID: "ccaeba79-8f73-4867-90d4-440328f2494a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:38:33.967573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:33.967460    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccaeba79-8f73-4867-90d4-440328f2494a-kube-api-access-mwmq7" (OuterVolumeSpecName: "kube-api-access-mwmq7") pod "ccaeba79-8f73-4867-90d4-440328f2494a" (UID: "ccaeba79-8f73-4867-90d4-440328f2494a"). InnerVolumeSpecName "kube-api-access-mwmq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:38:34.067001 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.066969    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmq7\" (UniqueName: \"kubernetes.io/projected/ccaeba79-8f73-4867-90d4-440328f2494a-kube-api-access-mwmq7\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:38:34.067001 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.066995    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccaeba79-8f73-4867-90d4-440328f2494a-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:38:34.067001 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.067005    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccaeba79-8f73-4867-90d4-440328f2494a-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:38:34.175411 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.175316    2554 generic.go:356] "Generic (PLEG): container finished" podID="ccaeba79-8f73-4867-90d4-440328f2494a" containerID="6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a" exitCode=0
Sep 10 17:38:34.175411 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.175356    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"
Sep 10 17:38:34.175633 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.175408    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv" event={"ID":"ccaeba79-8f73-4867-90d4-440328f2494a","Type":"ContainerDied","Data":"6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a"}
Sep 10 17:38:34.175633 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.175455    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv" event={"ID":"ccaeba79-8f73-4867-90d4-440328f2494a","Type":"ContainerDied","Data":"99c09ec450804e65e9817b9cec4903649d944d74f9eb235a5b57bb65f30882c3"}
Sep 10 17:38:34.175633 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.175477    2554 scope.go:117] "RemoveContainer" containerID="6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a"
Sep 10 17:38:34.184667 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.184645    2554 scope.go:117] "RemoveContainer" containerID="6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a"
Sep 10 17:38:34.184941 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:38:34.184921    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a\": container with ID starting with 6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a not found: ID does not exist" containerID="6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a"
Sep 10 17:38:34.185011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.184948    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a"} err="failed to get container status \"6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a\": rpc error: code = NotFound desc = could not find container \"6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a\": container with ID starting with 6fc2053e6daa690cc065ba1c740b05324ff971155498f13571df1de63a6c5a9a not found: ID does not exist"
Sep 10 17:38:34.197416 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.197381    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"]
Sep 10 17:38:34.200934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:34.200912    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/auth-test-model-3c30a2-kserve-7f97cd7bc-cb4kv"]
Sep 10 17:38:35.114573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:35.114521    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccaeba79-8f73-4867-90d4-440328f2494a" path="/var/lib/kubelet/pods/ccaeba79-8f73-4867-90d4-440328f2494a/volumes"
Sep 10 17:38:38.739791 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.739756    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"]
Sep 10 17:38:38.740182 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.740150    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccaeba79-8f73-4867-90d4-440328f2494a" containerName="main"
Sep 10 17:38:38.740237 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.740185    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccaeba79-8f73-4867-90d4-440328f2494a" containerName="main"
Sep 10 17:38:38.740273 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.740267    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccaeba79-8f73-4867-90d4-440328f2494a" containerName="main"
Sep 10 17:38:38.744871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.744853    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.748244 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.748226    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"default-dockercfg-r5g7h\""
Sep 10 17:38:38.749335 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.749316    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"openshift-service-ca.crt\""
Sep 10 17:38:38.749417 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.749381    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"sub-test-model-3c30a2-kserve-self-signed-certs\""
Sep 10 17:38:38.749945 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.749932    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3c30a2\"/\"kube-root-ca.crt\""
Sep 10 17:38:38.754406 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.754383    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"]
Sep 10 17:38:38.812221 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.812193    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-model-cache\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.812395 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.812280    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-dshm\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.812395 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.812319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9t7\" (UniqueName: \"kubernetes.io/projected/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-kube-api-access-gg9t7\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.812395 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.812338    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tmp-dir\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.812621 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.812401    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-home\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.812621 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.812420    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tls-certs\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914041 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.913999    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-dshm\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914041 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914041    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9t7\" (UniqueName: \"kubernetes.io/projected/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-kube-api-access-gg9t7\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914284 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914061    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tmp-dir\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914284 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-home\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914284 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tls-certs\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914284 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914253    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-model-cache\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914585 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914556    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tmp-dir\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914738 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914585    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-home\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.914738 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.914619    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-model-cache\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.916413 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.916392    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-dshm\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.916625 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.916607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tls-certs\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:38.922846 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:38.922824    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9t7\" (UniqueName: \"kubernetes.io/projected/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-kube-api-access-gg9t7\") pod \"sub-test-model-3c30a2-kserve-8489ccd579-z8r8f\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") " pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:39.055333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:39.055309    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:39.328874 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:39.328780    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"]
Sep 10 17:38:39.331187 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:38:39.331158    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e9aa5a2_ef44_4a62_84a1_6ad48bb29e7a.slice/crio-6f96515db0e0039b86a63f90f91458748070062f027ae4c1086fde20cafa9f9f WatchSource:0}: Error finding container 6f96515db0e0039b86a63f90f91458748070062f027ae4c1086fde20cafa9f9f: Status 404 returned error can't find the container with id 6f96515db0e0039b86a63f90f91458748070062f027ae4c1086fde20cafa9f9f
Sep 10 17:38:40.205509 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:40.205468    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f" event={"ID":"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a","Type":"ContainerStarted","Data":"18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923"}
Sep 10 17:38:40.205918 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:40.205518    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f" event={"ID":"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a","Type":"ContainerStarted","Data":"6f96515db0e0039b86a63f90f91458748070062f027ae4c1086fde20cafa9f9f"}
Sep 10 17:38:40.226054 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:40.226009    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f" podStartSLOduration=2.225993795 podStartE2EDuration="2.225993795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:40.224506406 +0000 UTC m=+1515.580996806" watchObservedRunningTime="2026-09-10 17:38:40.225993795 +0000 UTC m=+1515.582484218"
Sep 10 17:38:44.279898 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:44.279862    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:38:47.764819 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:47.764738    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"]
Sep 10 17:38:47.768405 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:47.768388    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:47.774965 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:47.774943    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a45272-dockercfg-qt5pc\""
Sep 10 17:38:47.775557 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:47.775452    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a45272\""
Sep 10 17:38:47.783210 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:47.783185    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"]
Sep 10 17:38:47.909809 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:47.909775    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbc876-4ac9-41bc-963b-0f3f25a2e710-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") " pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:47.909975 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:47.909816    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5z8h\" (UniqueName: \"kubernetes.io/projected/94bbc876-4ac9-41bc-963b-0f3f25a2e710-kube-api-access-m5z8h\") pod \"payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") " pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:48.011300 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:48.011256    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbc876-4ac9-41bc-963b-0f3f25a2e710-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") " pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:48.011485 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:48.011319    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5z8h\" (UniqueName: \"kubernetes.io/projected/94bbc876-4ac9-41bc-963b-0f3f25a2e710-kube-api-access-m5z8h\") pod \"payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") " pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:48.012195 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:48.012161    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbc876-4ac9-41bc-963b-0f3f25a2e710-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") " pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:48.019462 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:48.019393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5z8h\" (UniqueName: \"kubernetes.io/projected/94bbc876-4ac9-41bc-963b-0f3f25a2e710-kube-api-access-m5z8h\") pod \"payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") " pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:48.079381 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:48.079345    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:48.359741 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:48.359711    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"]
Sep 10 17:38:48.361636 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:38:48.361608    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bbc876_4ac9_41bc_963b_0f3f25a2e710.slice/crio-2cd88814172555d9a5936bd76da66e3cb71f4f44558765b102748642c5463303 WatchSource:0}: Error finding container 2cd88814172555d9a5936bd76da66e3cb71f4f44558765b102748642c5463303: Status 404 returned error can't find the container with id 2cd88814172555d9a5936bd76da66e3cb71f4f44558765b102748642c5463303
Sep 10 17:38:49.056154 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.056114    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:49.056580 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.056204    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:49.069029 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.069004    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:49.248754 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.248716    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp" event={"ID":"94bbc876-4ac9-41bc-963b-0f3f25a2e710","Type":"ContainerStarted","Data":"e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6"}
Sep 10 17:38:49.248952 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.248773    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp" event={"ID":"94bbc876-4ac9-41bc-963b-0f3f25a2e710","Type":"ContainerStarted","Data":"2cd88814172555d9a5936bd76da66e3cb71f4f44558765b102748642c5463303"}
Sep 10 17:38:49.248952 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.248868    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:38:49.259912 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.259889    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:38:49.270748 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:49.270701    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp" podStartSLOduration=2.270682527 podStartE2EDuration="2.270682527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:49.266368685 +0000 UTC m=+1524.622859096" watchObservedRunningTime="2026-09-10 17:38:49.270682527 +0000 UTC m=+1524.627172931"
Sep 10 17:38:59.966799 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:38:59.966758    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:39:00.255643 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:00.255524    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:39:04.899794 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.899761    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"]
Sep 10 17:39:04.903446 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.903429    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:04.907070 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.907043    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"openshift-service-ca.crt\""
Sep 10 17:39:04.907200 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.907053    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"kube-root-ca.crt\""
Sep 10 17:39:04.907200 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.907098    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"sub-test-model-c1fb61-kserve-self-signed-certs\""
Sep 10 17:39:04.907200 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.907074    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"default-dockercfg-jc8b9\""
Sep 10 17:39:04.911011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.910989    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"]
Sep 10 17:39:04.975035 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.975005    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-model-cache\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:04.975216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.975052    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tmp-dir\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:04.975216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.975075    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-dshm\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:04.975216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.975125    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tls-certs\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:04.975216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.975163    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vn7h\" (UniqueName: \"kubernetes.io/projected/5b8d34f9-030f-4c21-ace0-e8f434b09efa-kube-api-access-4vn7h\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:04.975413 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:04.975253    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-home\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.076601 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.076562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-model-cache\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.076768 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.076620    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tmp-dir\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.076768 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.076651    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-dshm\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.076768 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.076694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tls-certs\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.076768 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.076743    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vn7h\" (UniqueName: \"kubernetes.io/projected/5b8d34f9-030f-4c21-ace0-e8f434b09efa-kube-api-access-4vn7h\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.077046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.076795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-home\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.077046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.077026    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-model-cache\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.077128 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.077110    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tmp-dir\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.077168 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.077125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-home\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.078995 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.078973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-dshm\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.079142 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.079125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tls-certs\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.086006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.085982    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vn7h\" (UniqueName: \"kubernetes.io/projected/5b8d34f9-030f-4c21-ace0-e8f434b09efa-kube-api-access-4vn7h\") pod \"sub-test-model-c1fb61-kserve-657659c849-5cwgd\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.215201 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.215108    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:05.488081 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:05.488052    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"]
Sep 10 17:39:05.489989 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:39:05.489961    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b8d34f9_030f_4c21_ace0_e8f434b09efa.slice/crio-0e5260fbb208a2aebc45b630d4b369009b735f0bf44b3ea2520b47261151693c WatchSource:0}: Error finding container 0e5260fbb208a2aebc45b630d4b369009b735f0bf44b3ea2520b47261151693c: Status 404 returned error can't find the container with id 0e5260fbb208a2aebc45b630d4b369009b735f0bf44b3ea2520b47261151693c
Sep 10 17:39:06.337009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:06.336976    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd" event={"ID":"5b8d34f9-030f-4c21-ace0-e8f434b09efa","Type":"ContainerStarted","Data":"815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24"}
Sep 10 17:39:06.337009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:06.337015    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd" event={"ID":"5b8d34f9-030f-4c21-ace0-e8f434b09efa","Type":"ContainerStarted","Data":"0e5260fbb208a2aebc45b630d4b369009b735f0bf44b3ea2520b47261151693c"}
Sep 10 17:39:06.354229 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:06.354178    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd" podStartSLOduration=2.354161668 podStartE2EDuration="2.354161668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:39:06.35362276 +0000 UTC m=+1541.710113162" watchObservedRunningTime="2026-09-10 17:39:06.354161668 +0000 UTC m=+1541.710652067"
Sep 10 17:39:10.973226 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:10.973179    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:39:15.216014 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.215978    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:15.216467 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.216036    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:15.228142 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.228116    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:15.395215 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.395187    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:39:15.676479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.676438    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"]
Sep 10 17:39:15.680075 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.680059    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"
Sep 10 17:39:15.682425 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.682396    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:39:15.682566 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.682400    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:39:15.687307 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.687285    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"]
Sep 10 17:39:15.773214 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.773175    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385c7f13-97e3-4142-8cd8-0895fc3aa1fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq\" (UID: \"385c7f13-97e3-4142-8cd8-0895fc3aa1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"
Sep 10 17:39:15.874693 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.874660    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385c7f13-97e3-4142-8cd8-0895fc3aa1fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq\" (UID: \"385c7f13-97e3-4142-8cd8-0895fc3aa1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"
Sep 10 17:39:15.875313 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.875287    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385c7f13-97e3-4142-8cd8-0895fc3aa1fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq\" (UID: \"385c7f13-97e3-4142-8cd8-0895fc3aa1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"
Sep 10 17:39:15.992772 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:15.992690    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"
Sep 10 17:39:16.263798 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:16.263770    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"]
Sep 10 17:39:16.266521 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:39:16.266495    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385c7f13_97e3_4142_8cd8_0895fc3aa1fe.slice/crio-58eca2a5fb7a50a22655f405c5708536b753146a92ecc66bc2e54ec36deed1d5 WatchSource:0}: Error finding container 58eca2a5fb7a50a22655f405c5708536b753146a92ecc66bc2e54ec36deed1d5: Status 404 returned error can't find the container with id 58eca2a5fb7a50a22655f405c5708536b753146a92ecc66bc2e54ec36deed1d5
Sep 10 17:39:16.389742 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:16.389706    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq" event={"ID":"385c7f13-97e3-4142-8cd8-0895fc3aa1fe","Type":"ContainerStarted","Data":"58eca2a5fb7a50a22655f405c5708536b753146a92ecc66bc2e54ec36deed1d5"}
Sep 10 17:39:17.399445 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:17.399410    2554 generic.go:356] "Generic (PLEG): container finished" podID="385c7f13-97e3-4142-8cd8-0895fc3aa1fe" containerID="5c0b5e6ccbc445d7cc5900515b1147de6fce9bb0456187e1b02e565c6cc32baa" exitCode=0
Sep 10 17:39:17.399871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:17.399496    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq" event={"ID":"385c7f13-97e3-4142-8cd8-0895fc3aa1fe","Type":"ContainerDied","Data":"5c0b5e6ccbc445d7cc5900515b1147de6fce9bb0456187e1b02e565c6cc32baa"}
Sep 10 17:39:18.533624 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:18.533601    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"
Sep 10 17:39:18.602321 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:18.602289    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385c7f13-97e3-4142-8cd8-0895fc3aa1fe-maas-api-service-ca\") pod \"385c7f13-97e3-4142-8cd8-0895fc3aa1fe\" (UID: \"385c7f13-97e3-4142-8cd8-0895fc3aa1fe\") "
Sep 10 17:39:18.602719 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:18.602688    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385c7f13-97e3-4142-8cd8-0895fc3aa1fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "385c7f13-97e3-4142-8cd8-0895fc3aa1fe" (UID: "385c7f13-97e3-4142-8cd8-0895fc3aa1fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:18.703283 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:18.703207    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/385c7f13-97e3-4142-8cd8-0895fc3aa1fe-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:39:19.411554 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:19.411452    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"
Sep 10 17:39:19.411554 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:19.411454    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq" event={"ID":"385c7f13-97e3-4142-8cd8-0895fc3aa1fe","Type":"ContainerDied","Data":"58eca2a5fb7a50a22655f405c5708536b753146a92ecc66bc2e54ec36deed1d5"}
Sep 10 17:39:19.411554 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:19.411513    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58eca2a5fb7a50a22655f405c5708536b753146a92ecc66bc2e54ec36deed1d5"
Sep 10 17:39:31.533689 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.533655    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"]
Sep 10 17:39:31.534142 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.533888    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k" podUID="5e2659b0-336e-4b43-9998-bb05f275bb09" containerName="payload-pre-processing" containerID="cri-o://815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528" gracePeriod=30
Sep 10 17:39:31.703004 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.702963    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"]
Sep 10 17:39:31.705916 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.705887    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b5f818-kgzkq"]
Sep 10 17:39:31.785753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.785683    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:39:31.928279 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.928231    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2659b0-336e-4b43-9998-bb05f275bb09-plugins-config-volume\") pod \"5e2659b0-336e-4b43-9998-bb05f275bb09\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") "
Sep 10 17:39:31.928463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.928342    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxn4r\" (UniqueName: \"kubernetes.io/projected/5e2659b0-336e-4b43-9998-bb05f275bb09-kube-api-access-wxn4r\") pod \"5e2659b0-336e-4b43-9998-bb05f275bb09\" (UID: \"5e2659b0-336e-4b43-9998-bb05f275bb09\") "
Sep 10 17:39:31.928624 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.928602    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2659b0-336e-4b43-9998-bb05f275bb09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e2659b0-336e-4b43-9998-bb05f275bb09" (UID: "5e2659b0-336e-4b43-9998-bb05f275bb09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:31.930360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:31.930329    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2659b0-336e-4b43-9998-bb05f275bb09-kube-api-access-wxn4r" (OuterVolumeSpecName: "kube-api-access-wxn4r") pod "5e2659b0-336e-4b43-9998-bb05f275bb09" (UID: "5e2659b0-336e-4b43-9998-bb05f275bb09"). InnerVolumeSpecName "kube-api-access-wxn4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:39:32.029542 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.029491    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxn4r\" (UniqueName: \"kubernetes.io/projected/5e2659b0-336e-4b43-9998-bb05f275bb09-kube-api-access-wxn4r\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:39:32.029542 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.029524    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2659b0-336e-4b43-9998-bb05f275bb09-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:39:32.484304 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.484268    2554 generic.go:356] "Generic (PLEG): container finished" podID="5e2659b0-336e-4b43-9998-bb05f275bb09" containerID="815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528" exitCode=0
Sep 10 17:39:32.484503 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.484330    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"
Sep 10 17:39:32.484503 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.484361    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k" event={"ID":"5e2659b0-336e-4b43-9998-bb05f275bb09","Type":"ContainerDied","Data":"815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528"}
Sep 10 17:39:32.484503 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.484449    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k" event={"ID":"5e2659b0-336e-4b43-9998-bb05f275bb09","Type":"ContainerDied","Data":"95ea4dc7e769b8dc6c35d1299ea73f22f2a41302e1d83b7d0c2a33ac7261121a"}
Sep 10 17:39:32.484503 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.484477    2554 scope.go:117] "RemoveContainer" containerID="815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528"
Sep 10 17:39:32.493169 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.493150    2554 scope.go:117] "RemoveContainer" containerID="815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528"
Sep 10 17:39:32.493431 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:39:32.493413    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528\": container with ID starting with 815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528 not found: ID does not exist" containerID="815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528"
Sep 10 17:39:32.493489 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.493436    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528"} err="failed to get container status \"815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528\": rpc error: code = NotFound desc = could not find container \"815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528\": container with ID starting with 815158da2f2cc69f6aa79681cefd15de4f566259ef6905cdf0c9c83d54253528 not found: ID does not exist"
Sep 10 17:39:32.507103 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.507071    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"]
Sep 10 17:39:32.512520 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:32.512500    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b5f818-668b7c7bdb-cbg4k"]
Sep 10 17:39:33.114905 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:33.114874    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385c7f13-97e3-4142-8cd8-0895fc3aa1fe" path="/var/lib/kubelet/pods/385c7f13-97e3-4142-8cd8-0895fc3aa1fe/volumes"
Sep 10 17:39:33.115324 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:33.115313    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2659b0-336e-4b43-9998-bb05f275bb09" path="/var/lib/kubelet/pods/5e2659b0-336e-4b43-9998-bb05f275bb09/volumes"
Sep 10 17:39:35.198400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:35.198366    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:39:39.642073 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.642033    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"]
Sep 10 17:39:39.642638 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.642619    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385c7f13-97e3-4142-8cd8-0895fc3aa1fe" containerName="revoke-keys"
Sep 10 17:39:39.642690 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.642642    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="385c7f13-97e3-4142-8cd8-0895fc3aa1fe" containerName="revoke-keys"
Sep 10 17:39:39.642690 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.642660    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2659b0-336e-4b43-9998-bb05f275bb09" containerName="payload-pre-processing"
Sep 10 17:39:39.642690 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.642671    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2659b0-336e-4b43-9998-bb05f275bb09" containerName="payload-pre-processing"
Sep 10 17:39:39.642819 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.642808    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="385c7f13-97e3-4142-8cd8-0895fc3aa1fe" containerName="revoke-keys"
Sep 10 17:39:39.642857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.642828    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2659b0-336e-4b43-9998-bb05f275bb09" containerName="payload-pre-processing"
Sep 10 17:39:39.647221 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.647205    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"
Sep 10 17:39:39.650239 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.650217    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:39:39.650361 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.650307    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:39:39.666682 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.666655    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"]
Sep 10 17:39:39.807666 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.807620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a839b05-8836-4acc-b381-2a6036930c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a45272-h7cn6\" (UID: \"5a839b05-8836-4acc-b381-2a6036930c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"
Sep 10 17:39:39.908401 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.908312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a839b05-8836-4acc-b381-2a6036930c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a45272-h7cn6\" (UID: \"5a839b05-8836-4acc-b381-2a6036930c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"
Sep 10 17:39:39.908948 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.908928    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a839b05-8836-4acc-b381-2a6036930c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a45272-h7cn6\" (UID: \"5a839b05-8836-4acc-b381-2a6036930c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"
Sep 10 17:39:39.957594 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:39.957566    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"
Sep 10 17:39:40.236743 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:40.236719    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"]
Sep 10 17:39:40.238376 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:39:40.238346    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a839b05_8836_4acc_b381_2a6036930c9d.slice/crio-016c7e8ebdd0486b7a6d22e1b0857d5efca8770e7b883bf8797d45120a57327e WatchSource:0}: Error finding container 016c7e8ebdd0486b7a6d22e1b0857d5efca8770e7b883bf8797d45120a57327e: Status 404 returned error can't find the container with id 016c7e8ebdd0486b7a6d22e1b0857d5efca8770e7b883bf8797d45120a57327e
Sep 10 17:39:40.527890 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:40.527804    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6" event={"ID":"5a839b05-8836-4acc-b381-2a6036930c9d","Type":"ContainerStarted","Data":"016c7e8ebdd0486b7a6d22e1b0857d5efca8770e7b883bf8797d45120a57327e"}
Sep 10 17:39:41.534596 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:41.534558    2554 generic.go:356] "Generic (PLEG): container finished" podID="5a839b05-8836-4acc-b381-2a6036930c9d" containerID="4f5a7c3a53b21cc0ae4a4ff565548339b2050cd1a9c21490cd0b86f2cdfd1ffc" exitCode=0
Sep 10 17:39:41.534952 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:41.534597    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6" event={"ID":"5a839b05-8836-4acc-b381-2a6036930c9d","Type":"ContainerDied","Data":"4f5a7c3a53b21cc0ae4a4ff565548339b2050cd1a9c21490cd0b86f2cdfd1ffc"}
Sep 10 17:39:42.682330 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:42.682300    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"
Sep 10 17:39:42.836789 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:42.836756    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a839b05-8836-4acc-b381-2a6036930c9d-maas-api-service-ca\") pod \"5a839b05-8836-4acc-b381-2a6036930c9d\" (UID: \"5a839b05-8836-4acc-b381-2a6036930c9d\") "
Sep 10 17:39:42.837123 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:42.837098    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a839b05-8836-4acc-b381-2a6036930c9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a839b05-8836-4acc-b381-2a6036930c9d" (UID: "5a839b05-8836-4acc-b381-2a6036930c9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:42.937892 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:42.937855    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a839b05-8836-4acc-b381-2a6036930c9d-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:39:43.546643 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:43.546618    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"
Sep 10 17:39:43.546803 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:43.546621    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6" event={"ID":"5a839b05-8836-4acc-b381-2a6036930c9d","Type":"ContainerDied","Data":"016c7e8ebdd0486b7a6d22e1b0857d5efca8770e7b883bf8797d45120a57327e"}
Sep 10 17:39:43.546803 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:43.546737    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016c7e8ebdd0486b7a6d22e1b0857d5efca8770e7b883bf8797d45120a57327e"
Sep 10 17:39:54.852366 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:54.852334    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"]
Sep 10 17:39:55.311564 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.311513    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"]
Sep 10 17:39:55.311800 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.311774    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp" podUID="94bbc876-4ac9-41bc-963b-0f3f25a2e710" containerName="payload-processing" containerID="cri-o://e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6" gracePeriod=30
Sep 10 17:39:55.559730 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.559704    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:39:55.609703 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.609611    2554 generic.go:356] "Generic (PLEG): container finished" podID="94bbc876-4ac9-41bc-963b-0f3f25a2e710" containerID="e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6" exitCode=0
Sep 10 17:39:55.609703 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.609661    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"
Sep 10 17:39:55.609703 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.609685    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp" event={"ID":"94bbc876-4ac9-41bc-963b-0f3f25a2e710","Type":"ContainerDied","Data":"e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6"}
Sep 10 17:39:55.609984 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.609759    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp" event={"ID":"94bbc876-4ac9-41bc-963b-0f3f25a2e710","Type":"ContainerDied","Data":"2cd88814172555d9a5936bd76da66e3cb71f4f44558765b102748642c5463303"}
Sep 10 17:39:55.609984 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.609781    2554 scope.go:117] "RemoveContainer" containerID="e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6"
Sep 10 17:39:55.616868 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.616843    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"]
Sep 10 17:39:55.619006 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.618984    2554 scope.go:117] "RemoveContainer" containerID="e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6"
Sep 10 17:39:55.619258 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:39:55.619232    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6\": container with ID starting with e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6 not found: ID does not exist" containerID="e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6"
Sep 10 17:39:55.619333 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.619261    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6"} err="failed to get container status \"e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6\": rpc error: code = NotFound desc = could not find container \"e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6\": container with ID starting with e9b4070ae1d404f82cdd32ada7bf89db352379c53f5286b653c72ee6ef7f6da6 not found: ID does not exist"
Sep 10 17:39:55.622222 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.622202    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a45272-h7cn6"]
Sep 10 17:39:55.658722 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.658698    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5z8h\" (UniqueName: \"kubernetes.io/projected/94bbc876-4ac9-41bc-963b-0f3f25a2e710-kube-api-access-m5z8h\") pod \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") "
Sep 10 17:39:55.658820 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.658734    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbc876-4ac9-41bc-963b-0f3f25a2e710-plugins-config-volume\") pod \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\" (UID: \"94bbc876-4ac9-41bc-963b-0f3f25a2e710\") "
Sep 10 17:39:55.659104 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.659084    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bbc876-4ac9-41bc-963b-0f3f25a2e710-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94bbc876-4ac9-41bc-963b-0f3f25a2e710" (UID: "94bbc876-4ac9-41bc-963b-0f3f25a2e710"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:55.660783 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.660761    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bbc876-4ac9-41bc-963b-0f3f25a2e710-kube-api-access-m5z8h" (OuterVolumeSpecName: "kube-api-access-m5z8h") pod "94bbc876-4ac9-41bc-963b-0f3f25a2e710" (UID: "94bbc876-4ac9-41bc-963b-0f3f25a2e710"). InnerVolumeSpecName "kube-api-access-m5z8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:39:55.760250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.760217    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5z8h\" (UniqueName: \"kubernetes.io/projected/94bbc876-4ac9-41bc-963b-0f3f25a2e710-kube-api-access-m5z8h\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:39:55.760250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.760246    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bbc876-4ac9-41bc-963b-0f3f25a2e710-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:39:55.940615 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.940585    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"]
Sep 10 17:39:55.944270 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:55.944238    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a45272-7b76db5867-wjkjp"]
Sep 10 17:39:56.917184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:56.917153    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"]
Sep 10 17:39:57.115485 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:57.115451    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a839b05-8836-4acc-b381-2a6036930c9d" path="/var/lib/kubelet/pods/5a839b05-8836-4acc-b381-2a6036930c9d/volumes"
Sep 10 17:39:57.115936 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:57.115920    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94bbc876-4ac9-41bc-963b-0f3f25a2e710" path="/var/lib/kubelet/pods/94bbc876-4ac9-41bc-963b-0f3f25a2e710/volumes"
Sep 10 17:39:57.783850 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:39:57.783809    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:40:02.940062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.940025    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"]
Sep 10 17:40:02.940408 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.940397    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bbc876-4ac9-41bc-963b-0f3f25a2e710" containerName="payload-processing"
Sep 10 17:40:02.940450 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.940409    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bbc876-4ac9-41bc-963b-0f3f25a2e710" containerName="payload-processing"
Sep 10 17:40:02.940450 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.940423    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a839b05-8836-4acc-b381-2a6036930c9d" containerName="revoke-keys"
Sep 10 17:40:02.940450 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.940428    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a839b05-8836-4acc-b381-2a6036930c9d" containerName="revoke-keys"
Sep 10 17:40:02.940568 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.940487    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bbc876-4ac9-41bc-963b-0f3f25a2e710" containerName="payload-processing"
Sep 10 17:40:02.940568 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.940498    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a839b05-8836-4acc-b381-2a6036930c9d" containerName="revoke-keys"
Sep 10 17:40:02.943804 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.943784    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"
Sep 10 17:40:02.947208 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.947187    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:40:02.947385 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.947372    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:40:02.954953 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:02.954919    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"]
Sep 10 17:40:03.030555 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:03.030510    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e3db1e-ae10-4bd8-b19c-8560be3fdd71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr\" (UID: \"96e3db1e-ae10-4bd8-b19c-8560be3fdd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"
Sep 10 17:40:03.131622 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:03.131521    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e3db1e-ae10-4bd8-b19c-8560be3fdd71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr\" (UID: \"96e3db1e-ae10-4bd8-b19c-8560be3fdd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"
Sep 10 17:40:03.132115 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:03.132093    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e3db1e-ae10-4bd8-b19c-8560be3fdd71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr\" (UID: \"96e3db1e-ae10-4bd8-b19c-8560be3fdd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"
Sep 10 17:40:03.254782 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:03.254744    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"
Sep 10 17:40:03.534760 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:03.534734    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"]
Sep 10 17:40:03.537486 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:40:03.537458    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e3db1e_ae10_4bd8_b19c_8560be3fdd71.slice/crio-f55ed0ce12d23b41bae3f534e551243e40d80351765f7e9f9a73b760eb12064d WatchSource:0}: Error finding container f55ed0ce12d23b41bae3f534e551243e40d80351765f7e9f9a73b760eb12064d: Status 404 returned error can't find the container with id f55ed0ce12d23b41bae3f534e551243e40d80351765f7e9f9a73b760eb12064d
Sep 10 17:40:03.651802 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:03.651769    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr" event={"ID":"96e3db1e-ae10-4bd8-b19c-8560be3fdd71","Type":"ContainerStarted","Data":"f55ed0ce12d23b41bae3f534e551243e40d80351765f7e9f9a73b760eb12064d"}
Sep 10 17:40:04.657883 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:04.657848    2554 generic.go:356] "Generic (PLEG): container finished" podID="96e3db1e-ae10-4bd8-b19c-8560be3fdd71" containerID="7db18157b1497de8ea8cf198d1f487e5c2a37afb4c30e5b07a3fc1f264247919" exitCode=0
Sep 10 17:40:04.658263 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:04.657888    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr" event={"ID":"96e3db1e-ae10-4bd8-b19c-8560be3fdd71","Type":"ContainerDied","Data":"7db18157b1497de8ea8cf198d1f487e5c2a37afb4c30e5b07a3fc1f264247919"}
Sep 10 17:40:05.794829 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:05.794802    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"
Sep 10 17:40:05.859965 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:05.859929    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e3db1e-ae10-4bd8-b19c-8560be3fdd71-maas-api-service-ca\") pod \"96e3db1e-ae10-4bd8-b19c-8560be3fdd71\" (UID: \"96e3db1e-ae10-4bd8-b19c-8560be3fdd71\") "
Sep 10 17:40:05.860246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:05.860220    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e3db1e-ae10-4bd8-b19c-8560be3fdd71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96e3db1e-ae10-4bd8-b19c-8560be3fdd71" (UID: "96e3db1e-ae10-4bd8-b19c-8560be3fdd71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:05.960954 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:05.960861    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e3db1e-ae10-4bd8-b19c-8560be3fdd71-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:06.669776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:06.669749    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"
Sep 10 17:40:06.669947 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:06.669785    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr" event={"ID":"96e3db1e-ae10-4bd8-b19c-8560be3fdd71","Type":"ContainerDied","Data":"f55ed0ce12d23b41bae3f534e551243e40d80351765f7e9f9a73b760eb12064d"}
Sep 10 17:40:06.669947 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:06.669831    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55ed0ce12d23b41bae3f534e551243e40d80351765f7e9f9a73b760eb12064d"
Sep 10 17:40:09.871498 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:09.871456    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f" podUID="9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" containerName="main" containerID="cri-o://18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923" gracePeriod=45
Sep 10 17:40:10.123045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.122986    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:40:10.198606 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.198576    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tls-certs\") pod \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") "
Sep 10 17:40:10.198750 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.198620    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tmp-dir\") pod \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") "
Sep 10 17:40:10.198750 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.198642    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg9t7\" (UniqueName: \"kubernetes.io/projected/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-kube-api-access-gg9t7\") pod \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") "
Sep 10 17:40:10.198750 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.198742    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-model-cache\") pod \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") "
Sep 10 17:40:10.198890 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.198777    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-home\") pod \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") "
Sep 10 17:40:10.198890 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.198792    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-dshm\") pod \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\" (UID: \"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a\") "
Sep 10 17:40:10.198890 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.198875    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" (UID: "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:10.199065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.199041    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" (UID: "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:10.199130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.199060    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-home" (OuterVolumeSpecName: "home") pod "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" (UID: "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:10.199229 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.199212    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:10.199286 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.199234    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:10.199286 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.199248    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:10.200807 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.200785    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-dshm" (OuterVolumeSpecName: "dshm") pod "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" (UID: "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:10.201255 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.201233    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" (UID: "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:40:10.201255 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.201245    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-kube-api-access-gg9t7" (OuterVolumeSpecName: "kube-api-access-gg9t7") pod "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" (UID: "9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a"). InnerVolumeSpecName "kube-api-access-gg9t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:10.300145 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.300118    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:10.300145 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.300141    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:10.300145 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.300151    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg9t7\" (UniqueName: \"kubernetes.io/projected/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a-kube-api-access-gg9t7\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:10.691918 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.691882    2554 generic.go:356] "Generic (PLEG): container finished" podID="9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" containerID="18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923" exitCode=0
Sep 10 17:40:10.692084 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.691932    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"
Sep 10 17:40:10.692084 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.691964    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f" event={"ID":"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a","Type":"ContainerDied","Data":"18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923"}
Sep 10 17:40:10.692084 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.692007    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f" event={"ID":"9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a","Type":"ContainerDied","Data":"6f96515db0e0039b86a63f90f91458748070062f027ae4c1086fde20cafa9f9f"}
Sep 10 17:40:10.692084 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.692023    2554 scope.go:117] "RemoveContainer" containerID="18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923"
Sep 10 17:40:10.708376 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.708355    2554 scope.go:117] "RemoveContainer" containerID="18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923"
Sep 10 17:40:10.708654 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:40:10.708635    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923\": container with ID starting with 18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923 not found: ID does not exist" containerID="18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923"
Sep 10 17:40:10.708729 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.708666    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923"} err="failed to get container status \"18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923\": rpc error: code = NotFound desc = could not find container \"18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923\": container with ID starting with 18229c4d1c8f65ac22063a42974c91724b3c9a5c2d1d1605c9cf2988e513d923 not found: ID does not exist"
Sep 10 17:40:10.718048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.718024    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"]
Sep 10 17:40:10.721412 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:10.721376    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3c30a2/sub-test-model-3c30a2-kserve-8489ccd579-z8r8f"]
Sep 10 17:40:11.114344 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:11.114310    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" path="/var/lib/kubelet/pods/9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a/volumes"
Sep 10 17:40:11.936579 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:11.936517    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd" podUID="5b8d34f9-030f-4c21-ace0-e8f434b09efa" containerName="main" containerID="cri-o://815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24" gracePeriod=45
Sep 10 17:40:12.189927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.189865    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:40:12.317993 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.317944    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tmp-dir\") pod \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") "
Sep 10 17:40:12.318184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318014    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-home\") pod \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") "
Sep 10 17:40:12.318184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318035    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vn7h\" (UniqueName: \"kubernetes.io/projected/5b8d34f9-030f-4c21-ace0-e8f434b09efa-kube-api-access-4vn7h\") pod \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") "
Sep 10 17:40:12.318184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318053    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-dshm\") pod \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") "
Sep 10 17:40:12.318184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318089    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-model-cache\") pod \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") "
Sep 10 17:40:12.318184 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318125    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tls-certs\") pod \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\" (UID: \"5b8d34f9-030f-4c21-ace0-e8f434b09efa\") "
Sep 10 17:40:12.318469 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318280    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b8d34f9-030f-4c21-ace0-e8f434b09efa" (UID: "5b8d34f9-030f-4c21-ace0-e8f434b09efa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:12.318469 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318305    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-home" (OuterVolumeSpecName: "home") pod "5b8d34f9-030f-4c21-ace0-e8f434b09efa" (UID: "5b8d34f9-030f-4c21-ace0-e8f434b09efa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:12.318469 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318447    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b8d34f9-030f-4c21-ace0-e8f434b09efa" (UID: "5b8d34f9-030f-4c21-ace0-e8f434b09efa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:12.318653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318607    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:12.318653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318623    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:12.318653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.318638    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:12.320406 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.320361    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-dshm" (OuterVolumeSpecName: "dshm") pod "5b8d34f9-030f-4c21-ace0-e8f434b09efa" (UID: "5b8d34f9-030f-4c21-ace0-e8f434b09efa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:12.320494 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.320429    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b8d34f9-030f-4c21-ace0-e8f434b09efa" (UID: "5b8d34f9-030f-4c21-ace0-e8f434b09efa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:40:12.320494 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.320439    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8d34f9-030f-4c21-ace0-e8f434b09efa-kube-api-access-4vn7h" (OuterVolumeSpecName: "kube-api-access-4vn7h") pod "5b8d34f9-030f-4c21-ace0-e8f434b09efa" (UID: "5b8d34f9-030f-4c21-ace0-e8f434b09efa"). InnerVolumeSpecName "kube-api-access-4vn7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:12.420009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.419966    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vn7h\" (UniqueName: \"kubernetes.io/projected/5b8d34f9-030f-4c21-ace0-e8f434b09efa-kube-api-access-4vn7h\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:12.420009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.420003    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8d34f9-030f-4c21-ace0-e8f434b09efa-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:12.420009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.420015    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8d34f9-030f-4c21-ace0-e8f434b09efa-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:12.704067 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.704027    2554 generic.go:356] "Generic (PLEG): container finished" podID="5b8d34f9-030f-4c21-ace0-e8f434b09efa" containerID="815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24" exitCode=0
Sep 10 17:40:12.704246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.704072    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"
Sep 10 17:40:12.704246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.704116    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd" event={"ID":"5b8d34f9-030f-4c21-ace0-e8f434b09efa","Type":"ContainerDied","Data":"815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24"}
Sep 10 17:40:12.704246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.704164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd" event={"ID":"5b8d34f9-030f-4c21-ace0-e8f434b09efa","Type":"ContainerDied","Data":"0e5260fbb208a2aebc45b630d4b369009b735f0bf44b3ea2520b47261151693c"}
Sep 10 17:40:12.704246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.704185    2554 scope.go:117] "RemoveContainer" containerID="815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24"
Sep 10 17:40:12.713838 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.713823    2554 scope.go:117] "RemoveContainer" containerID="815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24"
Sep 10 17:40:12.714107 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:40:12.714089    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24\": container with ID starting with 815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24 not found: ID does not exist" containerID="815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24"
Sep 10 17:40:12.714156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.714117    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24"} err="failed to get container status \"815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24\": rpc error: code = NotFound desc = could not find container \"815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24\": container with ID starting with 815902e8f7d91c272247412049c31dcb505a6e0866f2dad17bc86bc788a13b24 not found: ID does not exist"
Sep 10 17:40:12.726351 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.726330    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"]
Sep 10 17:40:12.730046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:12.730024    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/sub-test-model-c1fb61-kserve-657659c849-5cwgd"]
Sep 10 17:40:13.114857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:13.114821    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8d34f9-030f-4c21-ace0-e8f434b09efa" path="/var/lib/kubelet/pods/5b8d34f9-030f-4c21-ace0-e8f434b09efa/volumes"
Sep 10 17:40:13.787815 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:13.787780    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"]
Sep 10 17:40:13.792776 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:13.792750    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-52f72f-zsngr"]
Sep 10 17:40:15.114147 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:15.114110    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96e3db1e-ae10-4bd8-b19c-8560be3fdd71" path="/var/lib/kubelet/pods/96e3db1e-ae10-4bd8-b19c-8560be3fdd71/volumes"
Sep 10 17:40:18.069628 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:18.069545    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:40:21.201770 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.201738    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"]
Sep 10 17:40:21.202167 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202120    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" containerName="main"
Sep 10 17:40:21.202167 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202132    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" containerName="main"
Sep 10 17:40:21.202167 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202140    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96e3db1e-ae10-4bd8-b19c-8560be3fdd71" containerName="revoke-keys"
Sep 10 17:40:21.202167 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202145    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="96e3db1e-ae10-4bd8-b19c-8560be3fdd71" containerName="revoke-keys"
Sep 10 17:40:21.202167 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202159    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8d34f9-030f-4c21-ace0-e8f434b09efa" containerName="main"
Sep 10 17:40:21.202167 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202165    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8d34f9-030f-4c21-ace0-e8f434b09efa" containerName="main"
Sep 10 17:40:21.202360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202236    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9aa5a2-ef44-4a62-84a1-6ad48bb29e7a" containerName="main"
Sep 10 17:40:21.202360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202247    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="96e3db1e-ae10-4bd8-b19c-8560be3fdd71" containerName="revoke-keys"
Sep 10 17:40:21.202360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.202255    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8d34f9-030f-4c21-ace0-e8f434b09efa" containerName="main"
Sep 10 17:40:21.206700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.206684    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"
Sep 10 17:40:21.209430 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.209397    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:40:21.209563 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.209513    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:40:21.215079 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.215053    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"]
Sep 10 17:40:21.303170 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.303132    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw\" (UID: \"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"
Sep 10 17:40:21.403996 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.403961    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw\" (UID: \"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"
Sep 10 17:40:21.404612 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.404595    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw\" (UID: \"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"
Sep 10 17:40:21.517732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.517657    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"
Sep 10 17:40:21.788108 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:21.788030    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"]
Sep 10 17:40:21.791508 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:40:21.791484    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621eb953_f7fa_4ad3_ae2e_3fa23e7e39d8.slice/crio-e410b268290b1235e7028e5911965ffc26d8397904186a353309a52a80868108 WatchSource:0}: Error finding container e410b268290b1235e7028e5911965ffc26d8397904186a353309a52a80868108: Status 404 returned error can't find the container with id e410b268290b1235e7028e5911965ffc26d8397904186a353309a52a80868108
Sep 10 17:40:22.757271 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:22.757172    2554 generic.go:356] "Generic (PLEG): container finished" podID="621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8" containerID="0fe26bcb21597f0e188d58816cabc7751ddf72e9b2ce07837f3e1404e218b0f3" exitCode=0
Sep 10 17:40:22.757271 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:22.757254    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw" event={"ID":"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8","Type":"ContainerDied","Data":"0fe26bcb21597f0e188d58816cabc7751ddf72e9b2ce07837f3e1404e218b0f3"}
Sep 10 17:40:22.757686 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:22.757306    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw" event={"ID":"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8","Type":"ContainerStarted","Data":"e410b268290b1235e7028e5911965ffc26d8397904186a353309a52a80868108"}
Sep 10 17:40:23.891181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:23.891156    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"
Sep 10 17:40:24.028604 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:24.028497    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8-maas-api-service-ca\") pod \"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8\" (UID: \"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8\") "
Sep 10 17:40:24.028870 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:24.028843    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8" (UID: "621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:24.130001 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:24.129967    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:24.768662 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:24.768625    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw" event={"ID":"621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8","Type":"ContainerDied","Data":"e410b268290b1235e7028e5911965ffc26d8397904186a353309a52a80868108"}
Sep 10 17:40:24.768828 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:24.768674    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e410b268290b1235e7028e5911965ffc26d8397904186a353309a52a80868108"
Sep 10 17:40:24.768828 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:24.768637    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"
Sep 10 17:40:32.025941 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:32.025903    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"]
Sep 10 17:40:32.030152 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:32.030122    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5912a3-w88qw"]
Sep 10 17:40:33.114694 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:33.114660    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8" path="/var/lib/kubelet/pods/621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8/volumes"
Sep 10 17:40:35.666567 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:35.666521    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:40:38.845432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:38.845394    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"]
Sep 10 17:40:38.845962 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:38.845613    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy" containerID="cri-o://15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8" gracePeriod=30
Sep 10 17:40:44.093510 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.093476    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:40:44.217721 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217628    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-socket\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.217721 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217673    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-envoy\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.217721 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217696    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-credential-socket\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217758    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563b6a7a-2c9e-4a54-9e43-17854336886f-istiod-ca-cert\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217791    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-data\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217813    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bngk8\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-kube-api-access-bngk8\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217840    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-token\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217874    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-certs\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217913    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-podinfo\") pod \"563b6a7a-2c9e-4a54-9e43-17854336886f\" (UID: \"563b6a7a-2c9e-4a54-9e43-17854336886f\") "
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.217935    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:44.218011 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218005    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218067    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-data" (OuterVolumeSpecName: "istio-data") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218153    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563b6a7a-2c9e-4a54-9e43-17854336886f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218247    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218359    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218379    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-credential-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218395    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563b6a7a-2c9e-4a54-9e43-17854336886f-istiod-ca-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218409    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-data\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.218424 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.218421    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-workload-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.219925 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.219901    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:44.220038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.220015    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-kube-api-access-bngk8" (OuterVolumeSpecName: "kube-api-access-bngk8") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "kube-api-access-bngk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:44.220514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.220496    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:40:44.220514 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.220506    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-token" (OuterVolumeSpecName: "istio-token") pod "563b6a7a-2c9e-4a54-9e43-17854336886f" (UID: "563b6a7a-2c9e-4a54-9e43-17854336886f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:44.319634 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.319592    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bngk8\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-kube-api-access-bngk8\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.319634 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.319634    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.319634 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.319644    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-podinfo\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.319634 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.319652    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563b6a7a-2c9e-4a54-9e43-17854336886f-istio-envoy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:40:44.870138 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.870099    2554 generic.go:356] "Generic (PLEG): container finished" podID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerID="15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8" exitCode=0
Sep 10 17:40:44.870331 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.870150    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" event={"ID":"563b6a7a-2c9e-4a54-9e43-17854336886f","Type":"ContainerDied","Data":"15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8"}
Sep 10 17:40:44.870331 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.870153    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"
Sep 10 17:40:44.870331 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.870190    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k" event={"ID":"563b6a7a-2c9e-4a54-9e43-17854336886f","Type":"ContainerDied","Data":"65bfe1ba8c00c0934117b1907561c67bdb50581bc52fc5c9497ae5b8102c79fd"}
Sep 10 17:40:44.870331 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.870216    2554 scope.go:117] "RemoveContainer" containerID="15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8"
Sep 10 17:40:44.880131 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.880115    2554 scope.go:117] "RemoveContainer" containerID="2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878"
Sep 10 17:40:44.887857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.887837    2554 scope.go:117] "RemoveContainer" containerID="15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8"
Sep 10 17:40:44.888145 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:40:44.888124    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8\": container with ID starting with 15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8 not found: ID does not exist" containerID="15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8"
Sep 10 17:40:44.888227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.888157    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8"} err="failed to get container status \"15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8\": rpc error: code = NotFound desc = could not find container \"15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8\": container with ID starting with 15ad713894d4765db943b07d59f0effb47374621f87db8ad61f7ed55ddd930b8 not found: ID does not exist"
Sep 10 17:40:44.888227 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.888179    2554 scope.go:117] "RemoveContainer" containerID="2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878"
Sep 10 17:40:44.888426 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:40:44.888410    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878\": container with ID starting with 2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878 not found: ID does not exist" containerID="2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878"
Sep 10 17:40:44.888466 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.888433    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878"} err="failed to get container status \"2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878\": rpc error: code = NotFound desc = could not find container \"2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878\": container with ID starting with 2a1e8c6e319c55c62c5d298926b1928c8d94bb7ec3e3e54d98b5182fd1459878 not found: ID does not exist"
Sep 10 17:40:44.895187 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.895164    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"]
Sep 10 17:40:44.899305 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:44.899280    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5912a3-openshift-default-65d6d7594-dgq7k"]
Sep 10 17:40:45.116295 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:45.116261    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" path="/var/lib/kubelet/pods/563b6a7a-2c9e-4a54-9e43-17854336886f/volumes"
Sep 10 17:40:49.079092 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:40:49.079052    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:41:00.470314 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:41:00.470278    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:41:10.972014 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:41:10.971974    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:41:20.477163 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:41:20.477123    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:41:34.169545 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:41:34.169499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:41:59.381453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:41:59.381371    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:42:25.351087 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:25.351056    2554 scope.go:117] "RemoveContainer" containerID="6261c99e801ebdb4dee0b29f615a55a90abdb44a98b3dd71dcc26e975e72b8ee"
Sep 10 17:42:32.516667 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.516635    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"]
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.516981    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy"
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.516992    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy"
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.517000    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy"
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.517007    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy"
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.517017    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8" containerName="revoke-keys"
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.517022    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8" containerName="revoke-keys"
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.517092    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy"
Sep 10 17:42:32.517130 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.517098    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="621eb953-f7fa-4ad3-ae2e-3fa23e7e39d8" containerName="revoke-keys"
Sep 10 17:42:32.519967 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.519949    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.523400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.523377    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7858f0\"/\"openshift-service-ca.crt\""
Sep 10 17:42:32.523400 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.523397    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7858f0\"/\"kube-root-ca.crt\""
Sep 10 17:42:32.523617 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.523424    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7858f0\"/\"ipp-route-7858f0-kserve-self-signed-certs\""
Sep 10 17:42:32.523617 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.523472    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7858f0\"/\"default-dockercfg-l44wf\""
Sep 10 17:42:32.530252 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.530229    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"]
Sep 10 17:42:32.621403 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.621368    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24e214a-727a-43df-9855-5db536cbf684-tls-certs\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.621403 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.621403    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-tmp-dir\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.621646 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.621423    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-home\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.621646 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.621443    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-model-cache\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.621646 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.621589    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtq7\" (UniqueName: \"kubernetes.io/projected/d24e214a-727a-43df-9855-5db536cbf684-kube-api-access-lqtq7\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.621646 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.621613    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-dshm\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.722507 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.722469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtq7\" (UniqueName: \"kubernetes.io/projected/d24e214a-727a-43df-9855-5db536cbf684-kube-api-access-lqtq7\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.722507 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.722507    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-dshm\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.722732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.722598    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24e214a-727a-43df-9855-5db536cbf684-tls-certs\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.722732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.722625    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-tmp-dir\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.722732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.722648    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-home\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.722732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.722675    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-model-cache\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.723048 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.723025    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-model-cache\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.723112 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.723092    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-home\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.723164 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.723103    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-tmp-dir\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.724935 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.724908    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-dshm\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.725185 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.725167    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24e214a-727a-43df-9855-5db536cbf684-tls-certs\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.730517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.730494    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtq7\" (UniqueName: \"kubernetes.io/projected/d24e214a-727a-43df-9855-5db536cbf684-kube-api-access-lqtq7\") pod \"ipp-route-7858f0-kserve-b967b86c5-8kxg2\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") " pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:32.829716 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:32.829682    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:33.113290 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:42:33.113256    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd24e214a_727a_43df_9855_5db536cbf684.slice/crio-d1bd677ed61fcb4c0eeaddd5ce1c83bf346dc17bd3896f4978ab90c391f86199 WatchSource:0}: Error finding container d1bd677ed61fcb4c0eeaddd5ce1c83bf346dc17bd3896f4978ab90c391f86199: Status 404 returned error can't find the container with id d1bd677ed61fcb4c0eeaddd5ce1c83bf346dc17bd3896f4978ab90c391f86199
Sep 10 17:42:33.115197 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:33.115173    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"]
Sep 10 17:42:33.406806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:33.406713    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2" event={"ID":"d24e214a-727a-43df-9855-5db536cbf684","Type":"ContainerStarted","Data":"02ff794853884cf704958944a59923b8c5e29748e7cae7aae8dfe5933a842c79"}
Sep 10 17:42:33.406806 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:33.406757    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2" event={"ID":"d24e214a-727a-43df-9855-5db536cbf684","Type":"ContainerStarted","Data":"d1bd677ed61fcb4c0eeaddd5ce1c83bf346dc17bd3896f4978ab90c391f86199"}
Sep 10 17:42:33.427290 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:33.427246    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2" podStartSLOduration=1.4272310799999999 podStartE2EDuration="1.42723108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:42:33.424862447 +0000 UTC m=+1748.781352848" watchObservedRunningTime="2026-09-10 17:42:33.42723108 +0000 UTC m=+1748.783721499"
Sep 10 17:42:36.874510 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:36.874475    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:42:42.830078 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:42.830042    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:42.830461 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:42.830123    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:42.842758 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:42.842736    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:43.468576 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:43.468547    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:42:51.378627 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:42:51.378590    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:43:06.467757 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.467721    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"]
Sep 10 17:43:06.468207 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.468161    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="563b6a7a-2c9e-4a54-9e43-17854336886f" containerName="istio-proxy"
Sep 10 17:43:06.471238 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.471221    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:06.473938 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.473907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8a34df-dockercfg-p4qzq\""
Sep 10 17:43:06.474103 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.474084    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8a34df\""
Sep 10 17:43:06.484120 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.484090    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"]
Sep 10 17:43:06.515374 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.515343    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac5edab-3467-4360-be2f-8f450211fa78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:06.515570 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.515488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bglfp\" (UniqueName: \"kubernetes.io/projected/6ac5edab-3467-4360-be2f-8f450211fa78-kube-api-access-bglfp\") pod \"payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:06.616777 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.616736    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac5edab-3467-4360-be2f-8f450211fa78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:06.616992 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.616829    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bglfp\" (UniqueName: \"kubernetes.io/projected/6ac5edab-3467-4360-be2f-8f450211fa78-kube-api-access-bglfp\") pod \"payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:06.617503 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.617476    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac5edab-3467-4360-be2f-8f450211fa78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:06.626460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.626434    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bglfp\" (UniqueName: \"kubernetes.io/projected/6ac5edab-3467-4360-be2f-8f450211fa78-kube-api-access-bglfp\") pod \"payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:06.781762 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:06.781668    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:07.055328 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:07.055298    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"]
Sep 10 17:43:07.057298 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:43:07.057272    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac5edab_3467_4360_be2f_8f450211fa78.slice/crio-2c0ed765077477783bf5f6aa98da039b3b0d7ed9c55879a393f16e7d3d558e55 WatchSource:0}: Error finding container 2c0ed765077477783bf5f6aa98da039b3b0d7ed9c55879a393f16e7d3d558e55: Status 404 returned error can't find the container with id 2c0ed765077477783bf5f6aa98da039b3b0d7ed9c55879a393f16e7d3d558e55
Sep 10 17:43:07.577828 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:07.577786    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m" event={"ID":"6ac5edab-3467-4360-be2f-8f450211fa78","Type":"ContainerStarted","Data":"22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70"}
Sep 10 17:43:07.578288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:07.577882    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m" event={"ID":"6ac5edab-3467-4360-be2f-8f450211fa78","Type":"ContainerStarted","Data":"2c0ed765077477783bf5f6aa98da039b3b0d7ed9c55879a393f16e7d3d558e55"}
Sep 10 17:43:07.578288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:07.577920    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:07.602203 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:07.602153    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m" podStartSLOduration=1.602138158 podStartE2EDuration="1.602138158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:07.599708064 +0000 UTC m=+1782.956198465" watchObservedRunningTime="2026-09-10 17:43:07.602138158 +0000 UTC m=+1782.958628557"
Sep 10 17:43:11.870979 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:11.870945    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:43:18.585198 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:18.585118    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:25.147276 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:25.147247    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:43:25.151410 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:25.151385    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:43:33.075891 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.075856    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"]
Sep 10 17:43:33.079233 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.079215    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"
Sep 10 17:43:33.081927 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.081902    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:43:33.082063 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.081986    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:43:33.088375 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.088350    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"]
Sep 10 17:43:33.268273 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.268233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5082119-127a-4a1a-bcbe-7e5709e1d075-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2\" (UID: \"f5082119-127a-4a1a-bcbe-7e5709e1d075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"
Sep 10 17:43:33.369875 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.369784    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5082119-127a-4a1a-bcbe-7e5709e1d075-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2\" (UID: \"f5082119-127a-4a1a-bcbe-7e5709e1d075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"
Sep 10 17:43:33.370406 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.370385    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5082119-127a-4a1a-bcbe-7e5709e1d075-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2\" (UID: \"f5082119-127a-4a1a-bcbe-7e5709e1d075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"
Sep 10 17:43:33.390358 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.390339    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"
Sep 10 17:43:33.662800 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.662730    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"]
Sep 10 17:43:33.664994 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:43:33.664963    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5082119_127a_4a1a_bcbe_7e5709e1d075.slice/crio-98fe7d29f720e7bb61b25154a9616a3020be5644ae8c485034692022c3dfb67d WatchSource:0}: Error finding container 98fe7d29f720e7bb61b25154a9616a3020be5644ae8c485034692022c3dfb67d: Status 404 returned error can't find the container with id 98fe7d29f720e7bb61b25154a9616a3020be5644ae8c485034692022c3dfb67d
Sep 10 17:43:33.708890 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:33.708865    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2" event={"ID":"f5082119-127a-4a1a-bcbe-7e5709e1d075","Type":"ContainerStarted","Data":"98fe7d29f720e7bb61b25154a9616a3020be5644ae8c485034692022c3dfb67d"}
Sep 10 17:43:34.716432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:34.716341    2554 generic.go:356] "Generic (PLEG): container finished" podID="f5082119-127a-4a1a-bcbe-7e5709e1d075" containerID="594bcfe678508ad13db3d0cbc28735a6dcf3a3f36205d6ce8926b98e59936978" exitCode=0
Sep 10 17:43:34.716432 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:34.716397    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2" event={"ID":"f5082119-127a-4a1a-bcbe-7e5709e1d075","Type":"ContainerDied","Data":"594bcfe678508ad13db3d0cbc28735a6dcf3a3f36205d6ce8926b98e59936978"}
Sep 10 17:43:35.852471 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:35.852443    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"
Sep 10 17:43:35.893986 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:35.893959    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5082119-127a-4a1a-bcbe-7e5709e1d075-maas-api-service-ca\") pod \"f5082119-127a-4a1a-bcbe-7e5709e1d075\" (UID: \"f5082119-127a-4a1a-bcbe-7e5709e1d075\") "
Sep 10 17:43:35.894343 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:35.894318    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5082119-127a-4a1a-bcbe-7e5709e1d075-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5082119-127a-4a1a-bcbe-7e5709e1d075" (UID: "f5082119-127a-4a1a-bcbe-7e5709e1d075"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:43:35.894419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:35.894403    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5082119-127a-4a1a-bcbe-7e5709e1d075-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:43:36.728349 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:36.728314    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2" event={"ID":"f5082119-127a-4a1a-bcbe-7e5709e1d075","Type":"ContainerDied","Data":"98fe7d29f720e7bb61b25154a9616a3020be5644ae8c485034692022c3dfb67d"}
Sep 10 17:43:36.728552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:36.728370    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98fe7d29f720e7bb61b25154a9616a3020be5644ae8c485034692022c3dfb67d"
Sep 10 17:43:36.728552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:36.728324    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"
Sep 10 17:43:44.081132 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.081095    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"]
Sep 10 17:43:44.081684 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.081362    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m" podUID="6ac5edab-3467-4360-be2f-8f450211fa78" containerName="payload-processing" containerID="cri-o://22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70" gracePeriod=30
Sep 10 17:43:44.274893 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.274857    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"]
Sep 10 17:43:44.278503 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.278474    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a34df-2jhx2"]
Sep 10 17:43:44.329035 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.329014    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:44.375671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.375580    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bglfp\" (UniqueName: \"kubernetes.io/projected/6ac5edab-3467-4360-be2f-8f450211fa78-kube-api-access-bglfp\") pod \"6ac5edab-3467-4360-be2f-8f450211fa78\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") "
Sep 10 17:43:44.375818 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.375726    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac5edab-3467-4360-be2f-8f450211fa78-plugins-config-volume\") pod \"6ac5edab-3467-4360-be2f-8f450211fa78\" (UID: \"6ac5edab-3467-4360-be2f-8f450211fa78\") "
Sep 10 17:43:44.376115 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.376090    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac5edab-3467-4360-be2f-8f450211fa78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ac5edab-3467-4360-be2f-8f450211fa78" (UID: "6ac5edab-3467-4360-be2f-8f450211fa78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:43:44.377625 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.377599    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac5edab-3467-4360-be2f-8f450211fa78-kube-api-access-bglfp" (OuterVolumeSpecName: "kube-api-access-bglfp") pod "6ac5edab-3467-4360-be2f-8f450211fa78" (UID: "6ac5edab-3467-4360-be2f-8f450211fa78"). InnerVolumeSpecName "kube-api-access-bglfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:43:44.476751 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.476720    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bglfp\" (UniqueName: \"kubernetes.io/projected/6ac5edab-3467-4360-be2f-8f450211fa78-kube-api-access-bglfp\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:43:44.476751 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.476748    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac5edab-3467-4360-be2f-8f450211fa78-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:43:44.773910 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.773825    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ac5edab-3467-4360-be2f-8f450211fa78" containerID="22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70" exitCode=0
Sep 10 17:43:44.773910 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.773868    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"
Sep 10 17:43:44.774102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.773870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m" event={"ID":"6ac5edab-3467-4360-be2f-8f450211fa78","Type":"ContainerDied","Data":"22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70"}
Sep 10 17:43:44.774102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.774010    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m" event={"ID":"6ac5edab-3467-4360-be2f-8f450211fa78","Type":"ContainerDied","Data":"2c0ed765077477783bf5f6aa98da039b3b0d7ed9c55879a393f16e7d3d558e55"}
Sep 10 17:43:44.774102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.774040    2554 scope.go:117] "RemoveContainer" containerID="22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70"
Sep 10 17:43:44.783391 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.783373    2554 scope.go:117] "RemoveContainer" containerID="22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70"
Sep 10 17:43:44.783703 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:43:44.783685    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70\": container with ID starting with 22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70 not found: ID does not exist" containerID="22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70"
Sep 10 17:43:44.783771 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.783711    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70"} err="failed to get container status \"22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70\": rpc error: code = NotFound desc = could not find container \"22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70\": container with ID starting with 22c78bb20b8a3c3ee68e7565a70a04b0b98892e393b786edf156d551f4778e70 not found: ID does not exist"
Sep 10 17:43:44.795988 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.795964    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"]
Sep 10 17:43:44.798213 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:44.798187    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a34df-79c7d5db87-hp97m"]
Sep 10 17:43:45.114734 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:45.114699    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac5edab-3467-4360-be2f-8f450211fa78" path="/var/lib/kubelet/pods/6ac5edab-3467-4360-be2f-8f450211fa78/volumes"
Sep 10 17:43:45.115111 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:45.115039    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5082119-127a-4a1a-bcbe-7e5709e1d075" path="/var/lib/kubelet/pods/f5082119-127a-4a1a-bcbe-7e5709e1d075/volumes"
Sep 10 17:43:48.868961 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:48.868926    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:43:54.696638 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.696597    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"]
Sep 10 17:43:54.697083 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.697065    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5082119-127a-4a1a-bcbe-7e5709e1d075" containerName="revoke-keys"
Sep 10 17:43:54.697083 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.697080    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5082119-127a-4a1a-bcbe-7e5709e1d075" containerName="revoke-keys"
Sep 10 17:43:54.697156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.697093    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac5edab-3467-4360-be2f-8f450211fa78" containerName="payload-processing"
Sep 10 17:43:54.697156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.697099    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac5edab-3467-4360-be2f-8f450211fa78" containerName="payload-processing"
Sep 10 17:43:54.697219 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.697172    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5082119-127a-4a1a-bcbe-7e5709e1d075" containerName="revoke-keys"
Sep 10 17:43:54.697219 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.697184    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac5edab-3467-4360-be2f-8f450211fa78" containerName="payload-processing"
Sep 10 17:43:54.701500 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.701483    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"
Sep 10 17:43:54.704225 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.704202    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:43:54.704338 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.704263    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:43:54.710013 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.709989    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"]
Sep 10 17:43:54.772816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.772778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cac73f-2558-4599-9700-2457ade4bc31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp\" (UID: \"63cac73f-2558-4599-9700-2457ade4bc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"
Sep 10 17:43:54.873948 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.873915    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cac73f-2558-4599-9700-2457ade4bc31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp\" (UID: \"63cac73f-2558-4599-9700-2457ade4bc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"
Sep 10 17:43:54.874563 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:54.874519    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cac73f-2558-4599-9700-2457ade4bc31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp\" (UID: \"63cac73f-2558-4599-9700-2457ade4bc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"
Sep 10 17:43:55.013179 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:55.013094    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"
Sep 10 17:43:55.282427 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:55.282399    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"]
Sep 10 17:43:55.284584 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:43:55.284557    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cac73f_2558_4599_9700_2457ade4bc31.slice/crio-8a2e049509b8836f676e923e4b2c21aa93bed4e9550fee0124b705e010ba4be2 WatchSource:0}: Error finding container 8a2e049509b8836f676e923e4b2c21aa93bed4e9550fee0124b705e010ba4be2: Status 404 returned error can't find the container with id 8a2e049509b8836f676e923e4b2c21aa93bed4e9550fee0124b705e010ba4be2
Sep 10 17:43:55.829188 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:55.829092    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp" event={"ID":"63cac73f-2558-4599-9700-2457ade4bc31","Type":"ContainerStarted","Data":"93e1a68adf1dfd0aca968f5c0394e691d375e4308807d3d8ca2875dc177af6d3"}
Sep 10 17:43:55.829188 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:55.829143    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp" event={"ID":"63cac73f-2558-4599-9700-2457ade4bc31","Type":"ContainerStarted","Data":"8a2e049509b8836f676e923e4b2c21aa93bed4e9550fee0124b705e010ba4be2"}
Sep 10 17:43:55.846538 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:55.846489    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp" podStartSLOduration=1.8464719889999999 podStartE2EDuration="1.846471989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:55.845340349 +0000 UTC m=+1831.201830760" watchObservedRunningTime="2026-09-10 17:43:55.846471989 +0000 UTC m=+1831.202962388"
Sep 10 17:43:56.836194 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:56.836160    2554 generic.go:356] "Generic (PLEG): container finished" podID="63cac73f-2558-4599-9700-2457ade4bc31" containerID="93e1a68adf1dfd0aca968f5c0394e691d375e4308807d3d8ca2875dc177af6d3" exitCode=0
Sep 10 17:43:56.836605 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:56.836241    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp" event={"ID":"63cac73f-2558-4599-9700-2457ade4bc31","Type":"ContainerDied","Data":"93e1a68adf1dfd0aca968f5c0394e691d375e4308807d3d8ca2875dc177af6d3"}
Sep 10 17:43:57.967248 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:57.967225    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"
Sep 10 17:43:58.100456 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:58.100354    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cac73f-2558-4599-9700-2457ade4bc31-maas-api-service-ca\") pod \"63cac73f-2558-4599-9700-2457ade4bc31\" (UID: \"63cac73f-2558-4599-9700-2457ade4bc31\") "
Sep 10 17:43:58.100826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:58.100795    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cac73f-2558-4599-9700-2457ade4bc31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63cac73f-2558-4599-9700-2457ade4bc31" (UID: "63cac73f-2558-4599-9700-2457ade4bc31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:43:58.201673 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:58.201638    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63cac73f-2558-4599-9700-2457ade4bc31-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:43:58.847566 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:58.847523    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"
Sep 10 17:43:58.847754 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:58.847515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp" event={"ID":"63cac73f-2558-4599-9700-2457ade4bc31","Type":"ContainerDied","Data":"8a2e049509b8836f676e923e4b2c21aa93bed4e9550fee0124b705e010ba4be2"}
Sep 10 17:43:58.847754 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:43:58.847659    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a2e049509b8836f676e923e4b2c21aa93bed4e9550fee0124b705e010ba4be2"
Sep 10 17:44:07.869678 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:07.869638    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:44:10.458676 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:10.458643    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"]
Sep 10 17:44:10.461177 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:10.461149    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7858f0-rqdrp"]
Sep 10 17:44:11.114603 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:11.114570    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63cac73f-2558-4599-9700-2457ade4bc31" path="/var/lib/kubelet/pods/63cac73f-2558-4599-9700-2457ade4bc31/volumes"
Sep 10 17:44:15.843260 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:15.843211    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"]
Sep 10 17:44:20.976518 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:20.976480    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:44:30.862818 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:30.862752    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2" podUID="d24e214a-727a-43df-9855-5db536cbf684" containerName="main" containerID="cri-o://02ff794853884cf704958944a59923b8c5e29748e7cae7aae8dfe5933a842c79" gracePeriod=45
Sep 10 17:44:31.009781 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.009740    2554 generic.go:356] "Generic (PLEG): container finished" podID="d24e214a-727a-43df-9855-5db536cbf684" containerID="02ff794853884cf704958944a59923b8c5e29748e7cae7aae8dfe5933a842c79" exitCode=0
Sep 10 17:44:31.009938 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.009806    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2" event={"ID":"d24e214a-727a-43df-9855-5db536cbf684","Type":"ContainerDied","Data":"02ff794853884cf704958944a59923b8c5e29748e7cae7aae8dfe5933a842c79"}
Sep 10 17:44:31.118648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.118585    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:44:31.213034 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213001    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-tmp-dir\") pod \"d24e214a-727a-43df-9855-5db536cbf684\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") "
Sep 10 17:44:31.213216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213050    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-home\") pod \"d24e214a-727a-43df-9855-5db536cbf684\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") "
Sep 10 17:44:31.213216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213078    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqtq7\" (UniqueName: \"kubernetes.io/projected/d24e214a-727a-43df-9855-5db536cbf684-kube-api-access-lqtq7\") pod \"d24e214a-727a-43df-9855-5db536cbf684\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") "
Sep 10 17:44:31.213216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213110    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-model-cache\") pod \"d24e214a-727a-43df-9855-5db536cbf684\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") "
Sep 10 17:44:31.213216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213144    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24e214a-727a-43df-9855-5db536cbf684-tls-certs\") pod \"d24e214a-727a-43df-9855-5db536cbf684\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") "
Sep 10 17:44:31.213216 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213162    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-dshm\") pod \"d24e214a-727a-43df-9855-5db536cbf684\" (UID: \"d24e214a-727a-43df-9855-5db536cbf684\") "
Sep 10 17:44:31.213489 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213280    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d24e214a-727a-43df-9855-5db536cbf684" (UID: "d24e214a-727a-43df-9855-5db536cbf684"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:31.213489 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213415    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-model-cache" (OuterVolumeSpecName: "model-cache") pod "d24e214a-727a-43df-9855-5db536cbf684" (UID: "d24e214a-727a-43df-9855-5db536cbf684"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:31.213489 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213422    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-home" (OuterVolumeSpecName: "home") pod "d24e214a-727a-43df-9855-5db536cbf684" (UID: "d24e214a-727a-43df-9855-5db536cbf684"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:31.213706 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213684    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:44:31.213770 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213709    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:44:31.213770 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.213722    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:44:31.215377 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.215343    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-dshm" (OuterVolumeSpecName: "dshm") pod "d24e214a-727a-43df-9855-5db536cbf684" (UID: "d24e214a-727a-43df-9855-5db536cbf684"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:31.215483 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.215406    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24e214a-727a-43df-9855-5db536cbf684-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d24e214a-727a-43df-9855-5db536cbf684" (UID: "d24e214a-727a-43df-9855-5db536cbf684"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:44:31.215545 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.215493    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24e214a-727a-43df-9855-5db536cbf684-kube-api-access-lqtq7" (OuterVolumeSpecName: "kube-api-access-lqtq7") pod "d24e214a-727a-43df-9855-5db536cbf684" (UID: "d24e214a-727a-43df-9855-5db536cbf684"). InnerVolumeSpecName "kube-api-access-lqtq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:44:31.315154 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.315117    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtq7\" (UniqueName: \"kubernetes.io/projected/d24e214a-727a-43df-9855-5db536cbf684-kube-api-access-lqtq7\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:44:31.315154 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.315150    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24e214a-727a-43df-9855-5db536cbf684-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:44:31.315154 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:31.315161    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24e214a-727a-43df-9855-5db536cbf684-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:44:32.017110 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:32.017069    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2" event={"ID":"d24e214a-727a-43df-9855-5db536cbf684","Type":"ContainerDied","Data":"d1bd677ed61fcb4c0eeaddd5ce1c83bf346dc17bd3896f4978ab90c391f86199"}
Sep 10 17:44:32.017110 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:32.017105    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"
Sep 10 17:44:32.017669 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:32.017139    2554 scope.go:117] "RemoveContainer" containerID="02ff794853884cf704958944a59923b8c5e29748e7cae7aae8dfe5933a842c79"
Sep 10 17:44:32.043780 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:32.043754    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"]
Sep 10 17:44:32.050572 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:32.050524    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7858f0/ipp-route-7858f0-kserve-b967b86c5-8kxg2"]
Sep 10 17:44:33.114245 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:33.114205    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24e214a-727a-43df-9855-5db536cbf684" path="/var/lib/kubelet/pods/d24e214a-727a-43df-9855-5db536cbf684/volumes"
Sep 10 17:44:59.443695 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.443657    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"]
Sep 10 17:44:59.446038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.444013    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63cac73f-2558-4599-9700-2457ade4bc31" containerName="revoke-keys"
Sep 10 17:44:59.446038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.444025    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="63cac73f-2558-4599-9700-2457ade4bc31" containerName="revoke-keys"
Sep 10 17:44:59.446038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.444040    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24e214a-727a-43df-9855-5db536cbf684" containerName="main"
Sep 10 17:44:59.446038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.444046    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24e214a-727a-43df-9855-5db536cbf684" containerName="main"
Sep 10 17:44:59.446038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.444101    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="63cac73f-2558-4599-9700-2457ade4bc31" containerName="revoke-keys"
Sep 10 17:44:59.446038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.444109    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24e214a-727a-43df-9855-5db536cbf684" containerName="main"
Sep 10 17:44:59.446875 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.446858    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"
Sep 10 17:44:59.449395 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.449374    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:44:59.449485 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.449374    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:44:59.455246 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.455222    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"]
Sep 10 17:44:59.494864 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.494832    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf8e90-98b8-4108-b1fa-83693c332f71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc\" (UID: \"c1cf8e90-98b8-4108-b1fa-83693c332f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"
Sep 10 17:44:59.595946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.595912    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf8e90-98b8-4108-b1fa-83693c332f71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc\" (UID: \"c1cf8e90-98b8-4108-b1fa-83693c332f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"
Sep 10 17:44:59.596552 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.596510    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf8e90-98b8-4108-b1fa-83693c332f71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc\" (UID: \"c1cf8e90-98b8-4108-b1fa-83693c332f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"
Sep 10 17:44:59.757995 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:44:59.757922    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"
Sep 10 17:45:00.030181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.030153    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"]
Sep 10 17:45:00.032072 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:45:00.032042    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cf8e90_98b8_4108_b1fa_83693c332f71.slice/crio-8b1a3952c26b40279de62d670c78e656e3f81fe54b633bc4d409334365302f8f WatchSource:0}: Error finding container 8b1a3952c26b40279de62d670c78e656e3f81fe54b633bc4d409334365302f8f: Status 404 returned error can't find the container with id 8b1a3952c26b40279de62d670c78e656e3f81fe54b633bc4d409334365302f8f
Sep 10 17:45:00.153422 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.153391    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"]
Sep 10 17:45:00.157802 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.157771    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"]
Sep 10 17:45:00.157946 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.157781    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"
Sep 10 17:45:00.161506 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.161484    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"
Sep 10 17:45:00.164448 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.164390    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc" event={"ID":"c1cf8e90-98b8-4108-b1fa-83693c332f71","Type":"ContainerStarted","Data":"8b1a3952c26b40279de62d670c78e656e3f81fe54b633bc4d409334365302f8f"}
Sep 10 17:45:00.169567 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.169429    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"]
Sep 10 17:45:00.172139 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.172113    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"]
Sep 10 17:45:00.172833 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.172813    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"
Sep 10 17:45:00.181499 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.181475    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"
Sep 10 17:45:00.453344 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.453311    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"]
Sep 10 17:45:00.455429 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:45:00.455403    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1ad0d1_c80a_47e4_883d_9c572752d803.slice/crio-06ea5abb63d0ee7520585ec2e1d3ed1729661ae75587cd1c375976ca8222e420 WatchSource:0}: Error finding container 06ea5abb63d0ee7520585ec2e1d3ed1729661ae75587cd1c375976ca8222e420: Status 404 returned error can't find the container with id 06ea5abb63d0ee7520585ec2e1d3ed1729661ae75587cd1c375976ca8222e420
Sep 10 17:45:00.463519 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:00.463301    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"]
Sep 10 17:45:00.467324 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:45:00.467299    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc2a366_8396_403f_8bcc_8515e9e121bf.slice/crio-4df0eae4230d5270948c8de19f783a1fc18e17ed70ee66ac3719e6646649ec02 WatchSource:0}: Error finding container 4df0eae4230d5270948c8de19f783a1fc18e17ed70ee66ac3719e6646649ec02: Status 404 returned error can't find the container with id 4df0eae4230d5270948c8de19f783a1fc18e17ed70ee66ac3719e6646649ec02
Sep 10 17:45:01.171123 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.171081    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh" event={"ID":"bcc2a366-8396-403f-8bcc-8515e9e121bf","Type":"ContainerStarted","Data":"f1873ef7dc620b0770934b354b26ddd354b7ac4ea683d9fc6150a1d159b9be40"}
Sep 10 17:45:01.171325 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.171142    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh" event={"ID":"bcc2a366-8396-403f-8bcc-8515e9e121bf","Type":"ContainerStarted","Data":"4df0eae4230d5270948c8de19f783a1fc18e17ed70ee66ac3719e6646649ec02"}
Sep 10 17:45:01.172585 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.172556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg" event={"ID":"7f1ad0d1-c80a-47e4-883d-9c572752d803","Type":"ContainerStarted","Data":"a89e310b9322f43d0969b9a37349a414fcfd8811168def3823e8a56195608575"}
Sep 10 17:45:01.172687 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.172610    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg" event={"ID":"7f1ad0d1-c80a-47e4-883d-9c572752d803","Type":"ContainerStarted","Data":"06ea5abb63d0ee7520585ec2e1d3ed1729661ae75587cd1c375976ca8222e420"}
Sep 10 17:45:01.174105 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.174078    2554 generic.go:356] "Generic (PLEG): container finished" podID="c1cf8e90-98b8-4108-b1fa-83693c332f71" containerID="10489d5815b22cfd1400620967a5301ec935aaa5c23e2fcbc9538acabc5b6fc6" exitCode=0
Sep 10 17:45:01.174202 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.174122    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc" event={"ID":"c1cf8e90-98b8-4108-b1fa-83693c332f71","Type":"ContainerDied","Data":"10489d5815b22cfd1400620967a5301ec935aaa5c23e2fcbc9538acabc5b6fc6"}
Sep 10 17:45:01.187250 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.187204    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh" podStartSLOduration=1.187190469 podStartE2EDuration="1.187190469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:01.18490104 +0000 UTC m=+1896.541391453" watchObservedRunningTime="2026-09-10 17:45:01.187190469 +0000 UTC m=+1896.543680868"
Sep 10 17:45:01.201394 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:01.201348    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg" podStartSLOduration=1.201334659 podStartE2EDuration="1.201334659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:01.199480795 +0000 UTC m=+1896.555971207" watchObservedRunningTime="2026-09-10 17:45:01.201334659 +0000 UTC m=+1896.557825050"
Sep 10 17:45:02.180700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.180667    2554 generic.go:356] "Generic (PLEG): container finished" podID="bcc2a366-8396-403f-8bcc-8515e9e121bf" containerID="f1873ef7dc620b0770934b354b26ddd354b7ac4ea683d9fc6150a1d159b9be40" exitCode=0
Sep 10 17:45:02.181120 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.180748    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh" event={"ID":"bcc2a366-8396-403f-8bcc-8515e9e121bf","Type":"ContainerDied","Data":"f1873ef7dc620b0770934b354b26ddd354b7ac4ea683d9fc6150a1d159b9be40"}
Sep 10 17:45:02.182292 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.182268    2554 generic.go:356] "Generic (PLEG): container finished" podID="7f1ad0d1-c80a-47e4-883d-9c572752d803" containerID="a89e310b9322f43d0969b9a37349a414fcfd8811168def3823e8a56195608575" exitCode=0
Sep 10 17:45:02.182374 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.182343    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg" event={"ID":"7f1ad0d1-c80a-47e4-883d-9c572752d803","Type":"ContainerDied","Data":"a89e310b9322f43d0969b9a37349a414fcfd8811168def3823e8a56195608575"}
Sep 10 17:45:02.313884 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.313859    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"
Sep 10 17:45:02.422586 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.422552    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf8e90-98b8-4108-b1fa-83693c332f71-maas-api-service-ca\") pod \"c1cf8e90-98b8-4108-b1fa-83693c332f71\" (UID: \"c1cf8e90-98b8-4108-b1fa-83693c332f71\") "
Sep 10 17:45:02.422982 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.422956    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cf8e90-98b8-4108-b1fa-83693c332f71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1cf8e90-98b8-4108-b1fa-83693c332f71" (UID: "c1cf8e90-98b8-4108-b1fa-83693c332f71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:02.523651 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:02.523564    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cf8e90-98b8-4108-b1fa-83693c332f71-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:03.188405 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:03.188376    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"
Sep 10 17:45:03.188894 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:03.188409    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc" event={"ID":"c1cf8e90-98b8-4108-b1fa-83693c332f71","Type":"ContainerDied","Data":"8b1a3952c26b40279de62d670c78e656e3f81fe54b633bc4d409334365302f8f"}
Sep 10 17:45:03.188894 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:03.188459    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b1a3952c26b40279de62d670c78e656e3f81fe54b633bc4d409334365302f8f"
Sep 10 17:45:03.323801 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:03.323775    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"
Sep 10 17:45:03.362999 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:03.362969    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"
Sep 10 17:45:04.195595 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:04.195557    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"
Sep 10 17:45:04.195595 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:04.195567    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh" event={"ID":"bcc2a366-8396-403f-8bcc-8515e9e121bf","Type":"ContainerDied","Data":"4df0eae4230d5270948c8de19f783a1fc18e17ed70ee66ac3719e6646649ec02"}
Sep 10 17:45:04.196062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:04.195627    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4df0eae4230d5270948c8de19f783a1fc18e17ed70ee66ac3719e6646649ec02"
Sep 10 17:45:04.196989 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:04.196967    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg" event={"ID":"7f1ad0d1-c80a-47e4-883d-9c572752d803","Type":"ContainerDied","Data":"06ea5abb63d0ee7520585ec2e1d3ed1729661ae75587cd1c375976ca8222e420"}
Sep 10 17:45:04.197124 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:04.197007    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ea5abb63d0ee7520585ec2e1d3ed1729661ae75587cd1c375976ca8222e420"
Sep 10 17:45:04.197124 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:04.196977    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"
Sep 10 17:45:09.971398 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:09.971363    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"]
Sep 10 17:45:09.973101 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:09.973080    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-fdb1bd-29817705-dxlrh"]
Sep 10 17:45:10.215149 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:10.215114    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"]
Sep 10 17:45:10.217163 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:10.217142    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fdb1bd-gzsnc"]
Sep 10 17:45:11.114511 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:11.114476    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc2a366-8396-403f-8bcc-8515e9e121bf" path="/var/lib/kubelet/pods/bcc2a366-8396-403f-8bcc-8515e9e121bf/volumes"
Sep 10 17:45:11.114878 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:11.114816    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1cf8e90-98b8-4108-b1fa-83693c332f71" path="/var/lib/kubelet/pods/c1cf8e90-98b8-4108-b1fa-83693c332f71/volumes"
Sep 10 17:45:13.473289 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:13.473256    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:45:17.189277 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189239    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"]
Sep 10 17:45:17.189687 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189675    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1ad0d1-c80a-47e4-883d-9c572752d803" containerName="cleanup"
Sep 10 17:45:17.189728 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189690    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1ad0d1-c80a-47e4-883d-9c572752d803" containerName="cleanup"
Sep 10 17:45:17.189728 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189703    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc2a366-8396-403f-8bcc-8515e9e121bf" containerName="cleanup"
Sep 10 17:45:17.189728 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189708    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc2a366-8396-403f-8bcc-8515e9e121bf" containerName="cleanup"
Sep 10 17:45:17.189728 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189723    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cf8e90-98b8-4108-b1fa-83693c332f71" containerName="revoke-keys"
Sep 10 17:45:17.189728 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189728    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cf8e90-98b8-4108-b1fa-83693c332f71" containerName="revoke-keys"
Sep 10 17:45:17.189881 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189784    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc2a366-8396-403f-8bcc-8515e9e121bf" containerName="cleanup"
Sep 10 17:45:17.189881 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189792    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f1ad0d1-c80a-47e4-883d-9c572752d803" containerName="cleanup"
Sep 10 17:45:17.189881 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.189801    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1cf8e90-98b8-4108-b1fa-83693c332f71" containerName="revoke-keys"
Sep 10 17:45:17.193231 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.193213    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"
Sep 10 17:45:17.197204 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.197182    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:45:17.197309 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.197248    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:45:17.204942 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.204920    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"]
Sep 10 17:45:17.373286 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.373237    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2866ffc4-962d-44e0-b671-c79a7d69117b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j\" (UID: \"2866ffc4-962d-44e0-b671-c79a7d69117b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"
Sep 10 17:45:17.474687 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.474590    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2866ffc4-962d-44e0-b671-c79a7d69117b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j\" (UID: \"2866ffc4-962d-44e0-b671-c79a7d69117b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"
Sep 10 17:45:17.475181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.475158    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2866ffc4-962d-44e0-b671-c79a7d69117b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j\" (UID: \"2866ffc4-962d-44e0-b671-c79a7d69117b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"
Sep 10 17:45:17.504094 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.504066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"
Sep 10 17:45:17.779211 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:17.778499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"]
Sep 10 17:45:17.782346 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:45:17.782316    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2866ffc4_962d_44e0_b671_c79a7d69117b.slice/crio-f565216f3754342de715dc8057bf68875ebaaa18bb28968861fe43d00b3ebd44 WatchSource:0}: Error finding container f565216f3754342de715dc8057bf68875ebaaa18bb28968861fe43d00b3ebd44: Status 404 returned error can't find the container with id f565216f3754342de715dc8057bf68875ebaaa18bb28968861fe43d00b3ebd44
Sep 10 17:45:18.273857 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:18.273825    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j" event={"ID":"2866ffc4-962d-44e0-b671-c79a7d69117b","Type":"ContainerStarted","Data":"8622611591b8cdb07397896cee12de9b01df979b3502e13f2d74f24d4379879c"}
Sep 10 17:45:18.274276 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:18.273877    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j" event={"ID":"2866ffc4-962d-44e0-b671-c79a7d69117b","Type":"ContainerStarted","Data":"f565216f3754342de715dc8057bf68875ebaaa18bb28968861fe43d00b3ebd44"}
Sep 10 17:45:18.289306 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:18.289256    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j" podStartSLOduration=1.289241423 podStartE2EDuration="1.289241423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:18.28833 +0000 UTC m=+1913.644820401" watchObservedRunningTime="2026-09-10 17:45:18.289241423 +0000 UTC m=+1913.645731823"
Sep 10 17:45:19.280783 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:19.280750    2554 generic.go:356] "Generic (PLEG): container finished" podID="2866ffc4-962d-44e0-b671-c79a7d69117b" containerID="8622611591b8cdb07397896cee12de9b01df979b3502e13f2d74f24d4379879c" exitCode=0
Sep 10 17:45:19.281151 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:19.280788    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j" event={"ID":"2866ffc4-962d-44e0-b671-c79a7d69117b","Type":"ContainerDied","Data":"8622611591b8cdb07397896cee12de9b01df979b3502e13f2d74f24d4379879c"}
Sep 10 17:45:20.431915 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:20.431890    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"
Sep 10 17:45:20.500738 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:20.500711    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2866ffc4-962d-44e0-b671-c79a7d69117b-maas-api-service-ca\") pod \"2866ffc4-962d-44e0-b671-c79a7d69117b\" (UID: \"2866ffc4-962d-44e0-b671-c79a7d69117b\") "
Sep 10 17:45:20.501028 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:20.501007    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2866ffc4-962d-44e0-b671-c79a7d69117b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2866ffc4-962d-44e0-b671-c79a7d69117b" (UID: "2866ffc4-962d-44e0-b671-c79a7d69117b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:20.601328 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:20.601300    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2866ffc4-962d-44e0-b671-c79a7d69117b-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:21.293257 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:21.293226    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"
Sep 10 17:45:21.293419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:21.293262    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j" event={"ID":"2866ffc4-962d-44e0-b671-c79a7d69117b","Type":"ContainerDied","Data":"f565216f3754342de715dc8057bf68875ebaaa18bb28968861fe43d00b3ebd44"}
Sep 10 17:45:21.293419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:21.293311    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f565216f3754342de715dc8057bf68875ebaaa18bb28968861fe43d00b3ebd44"
Sep 10 17:45:25.391795 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:25.391761    2554 scope.go:117] "RemoveContainer" containerID="5c0b5e6ccbc445d7cc5900515b1147de6fce9bb0456187e1b02e565c6cc32baa"
Sep 10 17:45:28.184299 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.184268    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"]
Sep 10 17:45:28.187815 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.187790    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3c30a2-29817705-zdgjg"]
Sep 10 17:45:28.211386 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.211355    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"]
Sep 10 17:45:28.211674 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.211616    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq" podUID="42fb6b85-280c-44c3-9391-298d3eb255f0" containerName="payload-processing" containerID="cri-o://ebae9337c6dc6dd8d273a8e60e4dcbedca34a6aafe2f31b2b31a975d1dfbd5ed" gracePeriod=30
Sep 10 17:45:28.221911 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.221884    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"]
Sep 10 17:45:28.222102 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.222081    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss" podUID="7c9acb33-d0d2-407d-8fff-cf20e943fee2" containerName="payload-pre-processing" containerID="cri-o://20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709" gracePeriod=30
Sep 10 17:45:28.329099 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.329073    2554 generic.go:356] "Generic (PLEG): container finished" podID="42fb6b85-280c-44c3-9391-298d3eb255f0" containerID="ebae9337c6dc6dd8d273a8e60e4dcbedca34a6aafe2f31b2b31a975d1dfbd5ed" exitCode=0
Sep 10 17:45:28.329185 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.329154    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq" event={"ID":"42fb6b85-280c-44c3-9391-298d3eb255f0","Type":"ContainerDied","Data":"ebae9337c6dc6dd8d273a8e60e4dcbedca34a6aafe2f31b2b31a975d1dfbd5ed"}
Sep 10 17:45:28.396181 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.396126    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"]
Sep 10 17:45:28.398847 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.398818    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3c30a2-4428j"]
Sep 10 17:45:28.489324 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.489299    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:45:28.492822 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.492803    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:45:28.575544 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.575502    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9acb33-d0d2-407d-8fff-cf20e943fee2-plugins-config-volume\") pod \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") "
Sep 10 17:45:28.575718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.575625    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fb6b85-280c-44c3-9391-298d3eb255f0-plugins-config-volume\") pod \"42fb6b85-280c-44c3-9391-298d3eb255f0\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") "
Sep 10 17:45:28.575718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.575691    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-526pk\" (UniqueName: \"kubernetes.io/projected/7c9acb33-d0d2-407d-8fff-cf20e943fee2-kube-api-access-526pk\") pod \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\" (UID: \"7c9acb33-d0d2-407d-8fff-cf20e943fee2\") "
Sep 10 17:45:28.575814 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.575744    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cs9j\" (UniqueName: \"kubernetes.io/projected/42fb6b85-280c-44c3-9391-298d3eb255f0-kube-api-access-4cs9j\") pod \"42fb6b85-280c-44c3-9391-298d3eb255f0\" (UID: \"42fb6b85-280c-44c3-9391-298d3eb255f0\") "
Sep 10 17:45:28.575879 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.575852    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9acb33-d0d2-407d-8fff-cf20e943fee2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c9acb33-d0d2-407d-8fff-cf20e943fee2" (UID: "7c9acb33-d0d2-407d-8fff-cf20e943fee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:28.576038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.576014    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb6b85-280c-44c3-9391-298d3eb255f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42fb6b85-280c-44c3-9391-298d3eb255f0" (UID: "42fb6b85-280c-44c3-9391-298d3eb255f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:28.576038 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.576029    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9acb33-d0d2-407d-8fff-cf20e943fee2-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:28.577800 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.577776    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fb6b85-280c-44c3-9391-298d3eb255f0-kube-api-access-4cs9j" (OuterVolumeSpecName: "kube-api-access-4cs9j") pod "42fb6b85-280c-44c3-9391-298d3eb255f0" (UID: "42fb6b85-280c-44c3-9391-298d3eb255f0"). InnerVolumeSpecName "kube-api-access-4cs9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:28.577888 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.577820    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9acb33-d0d2-407d-8fff-cf20e943fee2-kube-api-access-526pk" (OuterVolumeSpecName: "kube-api-access-526pk") pod "7c9acb33-d0d2-407d-8fff-cf20e943fee2" (UID: "7c9acb33-d0d2-407d-8fff-cf20e943fee2"). InnerVolumeSpecName "kube-api-access-526pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:28.677040 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.677009    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-526pk\" (UniqueName: \"kubernetes.io/projected/7c9acb33-d0d2-407d-8fff-cf20e943fee2-kube-api-access-526pk\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:28.677040 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.677037    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cs9j\" (UniqueName: \"kubernetes.io/projected/42fb6b85-280c-44c3-9391-298d3eb255f0-kube-api-access-4cs9j\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:28.677178 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:28.677047    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42fb6b85-280c-44c3-9391-298d3eb255f0-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:29.114426 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.114376    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2866ffc4-962d-44e0-b671-c79a7d69117b" path="/var/lib/kubelet/pods/2866ffc4-962d-44e0-b671-c79a7d69117b/volumes"
Sep 10 17:45:29.114818 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.114795    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f1ad0d1-c80a-47e4-883d-9c572752d803" path="/var/lib/kubelet/pods/7f1ad0d1-c80a-47e4-883d-9c572752d803/volumes"
Sep 10 17:45:29.335215 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.335175    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"
Sep 10 17:45:29.335700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.335174    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq" event={"ID":"42fb6b85-280c-44c3-9391-298d3eb255f0","Type":"ContainerDied","Data":"ca88a205c522f8e0be62c2f7c72c3986103fe9983e7da43428ea71acaba21e5e"}
Sep 10 17:45:29.335700 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.335331    2554 scope.go:117] "RemoveContainer" containerID="ebae9337c6dc6dd8d273a8e60e4dcbedca34a6aafe2f31b2b31a975d1dfbd5ed"
Sep 10 17:45:29.336687 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.336629    2554 generic.go:356] "Generic (PLEG): container finished" podID="7c9acb33-d0d2-407d-8fff-cf20e943fee2" containerID="20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709" exitCode=0
Sep 10 17:45:29.336780 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.336686    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"
Sep 10 17:45:29.336780 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.336761    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss" event={"ID":"7c9acb33-d0d2-407d-8fff-cf20e943fee2","Type":"ContainerDied","Data":"20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709"}
Sep 10 17:45:29.336892 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.336817    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss" event={"ID":"7c9acb33-d0d2-407d-8fff-cf20e943fee2","Type":"ContainerDied","Data":"6a02360474543d5cf6434a488f38668cd677d88b7e0db9d349ba7347d958ff99"}
Sep 10 17:45:29.344746 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.344671    2554 scope.go:117] "RemoveContainer" containerID="20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709"
Sep 10 17:45:29.352519 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.352478    2554 scope.go:117] "RemoveContainer" containerID="20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709"
Sep 10 17:45:29.352850 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:45:29.352819    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709\": container with ID starting with 20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709 not found: ID does not exist" containerID="20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709"
Sep 10 17:45:29.352924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.352852    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709"} err="failed to get container status \"20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709\": rpc error: code = NotFound desc = could not find container \"20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709\": container with ID starting with 20a19c999c6728ac5fd8a4ad85b198d4b214642420539b8b1f258bc4ec4ce709 not found: ID does not exist"
Sep 10 17:45:29.361272 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.361252    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"]
Sep 10 17:45:29.365269 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.365221    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3c30a2-7b4d4d674b-kpqss"]
Sep 10 17:45:29.377129 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.377106    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"]
Sep 10 17:45:29.385750 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:29.385730    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3c30a2-6dd9989579-zbvqq"]
Sep 10 17:45:31.114349 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:31.114314    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42fb6b85-280c-44c3-9391-298d3eb255f0" path="/var/lib/kubelet/pods/42fb6b85-280c-44c3-9391-298d3eb255f0/volumes"
Sep 10 17:45:31.114846 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:31.114829    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9acb33-d0d2-407d-8fff-cf20e943fee2" path="/var/lib/kubelet/pods/7c9acb33-d0d2-407d-8fff-cf20e943fee2/volumes"
Sep 10 17:45:31.269479 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:31.269437    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:45:36.031047 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031011    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"]
Sep 10 17:45:36.031413 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031381    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2866ffc4-962d-44e0-b671-c79a7d69117b" containerName="revoke-keys"
Sep 10 17:45:36.031413 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031393    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2866ffc4-962d-44e0-b671-c79a7d69117b" containerName="revoke-keys"
Sep 10 17:45:36.031413 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031411    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9acb33-d0d2-407d-8fff-cf20e943fee2" containerName="payload-pre-processing"
Sep 10 17:45:36.031517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031416    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9acb33-d0d2-407d-8fff-cf20e943fee2" containerName="payload-pre-processing"
Sep 10 17:45:36.031517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031422    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42fb6b85-280c-44c3-9391-298d3eb255f0" containerName="payload-processing"
Sep 10 17:45:36.031517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031428    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="42fb6b85-280c-44c3-9391-298d3eb255f0" containerName="payload-processing"
Sep 10 17:45:36.031517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031483    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="42fb6b85-280c-44c3-9391-298d3eb255f0" containerName="payload-processing"
Sep 10 17:45:36.031517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031494    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2866ffc4-962d-44e0-b671-c79a7d69117b" containerName="revoke-keys"
Sep 10 17:45:36.031517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.031501    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9acb33-d0d2-407d-8fff-cf20e943fee2" containerName="payload-pre-processing"
Sep 10 17:45:36.034441 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.034424    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"
Sep 10 17:45:36.037022 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.037001    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:45:36.037158 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.037135    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\""
Sep 10 17:45:36.042815 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.042792    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"]
Sep 10 17:45:36.143357 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.143326    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3fc3b-9194-43df-b6dd-d9d93cf32e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x\" (UID: \"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"
Sep 10 17:45:36.244070 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.244032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3fc3b-9194-43df-b6dd-d9d93cf32e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x\" (UID: \"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"
Sep 10 17:45:36.244686 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.244664    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3fc3b-9194-43df-b6dd-d9d93cf32e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x\" (UID: \"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"
Sep 10 17:45:36.345651 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.345607    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"
Sep 10 17:45:36.638023 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:36.637996    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"]
Sep 10 17:45:36.640146 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:45:36.640111    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f3fc3b_9194_43df_b6dd_d9d93cf32e45.slice/crio-2ea54252ccc338c55da4c6f205552a5c60ee5e044f4d5d2735693a70e41c529d WatchSource:0}: Error finding container 2ea54252ccc338c55da4c6f205552a5c60ee5e044f4d5d2735693a70e41c529d: Status 404 returned error can't find the container with id 2ea54252ccc338c55da4c6f205552a5c60ee5e044f4d5d2735693a70e41c529d
Sep 10 17:45:37.380936 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:37.380907    2554 generic.go:356] "Generic (PLEG): container finished" podID="c2f3fc3b-9194-43df-b6dd-d9d93cf32e45" containerID="9a7dbbc9fe55b97a606a5ff3dcce025a40d35e68809f46090126423b0b253b1b" exitCode=0
Sep 10 17:45:37.381319 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:37.380943    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x" event={"ID":"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45","Type":"ContainerDied","Data":"9a7dbbc9fe55b97a606a5ff3dcce025a40d35e68809f46090126423b0b253b1b"}
Sep 10 17:45:37.381319 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:37.380979    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x" event={"ID":"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45","Type":"ContainerStarted","Data":"2ea54252ccc338c55da4c6f205552a5c60ee5e044f4d5d2735693a70e41c529d"}
Sep 10 17:45:38.515305 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:38.515272    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"
Sep 10 17:45:38.565491 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:38.565459    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3fc3b-9194-43df-b6dd-d9d93cf32e45-maas-api-service-ca\") pod \"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45\" (UID: \"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45\") "
Sep 10 17:45:38.565783 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:38.565761    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f3fc3b-9194-43df-b6dd-d9d93cf32e45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2f3fc3b-9194-43df-b6dd-d9d93cf32e45" (UID: "c2f3fc3b-9194-43df-b6dd-d9d93cf32e45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:38.666880 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:38.666807    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3fc3b-9194-43df-b6dd-d9d93cf32e45-maas-api-service-ca\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:39.392080 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:39.391997    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"
Sep 10 17:45:39.392080 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:39.392045    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x" event={"ID":"c2f3fc3b-9194-43df-b6dd-d9d93cf32e45","Type":"ContainerDied","Data":"2ea54252ccc338c55da4c6f205552a5c60ee5e044f4d5d2735693a70e41c529d"}
Sep 10 17:45:39.392258 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:39.392091    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea54252ccc338c55da4c6f205552a5c60ee5e044f4d5d2735693a70e41c529d"
Sep 10 17:45:46.580710 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.580678    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"]
Sep 10 17:45:46.581177 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.580881    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7" podUID="c56cb3fe-3eaa-4340-9a51-3cff321c98b1" containerName="payload-processing" containerID="cri-o://f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb" gracePeriod=30
Sep 10 17:45:46.762198 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.762168    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"]
Sep 10 17:45:46.767464 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.767432    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c1fb61-qsr5x"]
Sep 10 17:45:46.844068 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.844005    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:45:46.938478 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.938439    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-plugins-config-volume\") pod \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") "
Sep 10 17:45:46.938678 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.938500    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlpcs\" (UniqueName: \"kubernetes.io/projected/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-kube-api-access-rlpcs\") pod \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\" (UID: \"c56cb3fe-3eaa-4340-9a51-3cff321c98b1\") "
Sep 10 17:45:46.938901 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.938877    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c56cb3fe-3eaa-4340-9a51-3cff321c98b1" (UID: "c56cb3fe-3eaa-4340-9a51-3cff321c98b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:46.940590 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:46.940562    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-kube-api-access-rlpcs" (OuterVolumeSpecName: "kube-api-access-rlpcs") pod "c56cb3fe-3eaa-4340-9a51-3cff321c98b1" (UID: "c56cb3fe-3eaa-4340-9a51-3cff321c98b1"). InnerVolumeSpecName "kube-api-access-rlpcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:47.040465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.040422    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-plugins-config-volume\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:47.040465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.040466    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlpcs\" (UniqueName: \"kubernetes.io/projected/c56cb3fe-3eaa-4340-9a51-3cff321c98b1-kube-api-access-rlpcs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:47.113777 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.113695    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2f3fc3b-9194-43df-b6dd-d9d93cf32e45" path="/var/lib/kubelet/pods/c2f3fc3b-9194-43df-b6dd-d9d93cf32e45/volumes"
Sep 10 17:45:47.433602 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.433501    2554 generic.go:356] "Generic (PLEG): container finished" podID="c56cb3fe-3eaa-4340-9a51-3cff321c98b1" containerID="f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb" exitCode=0
Sep 10 17:45:47.433602 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.433568    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"
Sep 10 17:45:47.433602 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.433589    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7" event={"ID":"c56cb3fe-3eaa-4340-9a51-3cff321c98b1","Type":"ContainerDied","Data":"f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb"}
Sep 10 17:45:47.433877 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.433650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7" event={"ID":"c56cb3fe-3eaa-4340-9a51-3cff321c98b1","Type":"ContainerDied","Data":"b4d2ba5d9b03fcd4bcd618016c039d682cf9c354f44141f7d330ea7429d368b6"}
Sep 10 17:45:47.433877 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.433680    2554 scope.go:117] "RemoveContainer" containerID="f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb"
Sep 10 17:45:47.442288 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.442269    2554 scope.go:117] "RemoveContainer" containerID="f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb"
Sep 10 17:45:47.442597 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:45:47.442570    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb\": container with ID starting with f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb not found: ID does not exist" containerID="f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb"
Sep 10 17:45:47.442718 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.442608    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb"} err="failed to get container status \"f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb\": rpc error: code = NotFound desc = could not find container \"f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb\": container with ID starting with f9b671e6ff5f1be75a82f6c3865e00600f2ac9c86cfdcf8eac858104c3f2e2cb not found: ID does not exist"
Sep 10 17:45:47.454826 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.454801    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"]
Sep 10 17:45:47.458881 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:47.458859    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c1fb61-6595dfd5cf-qmcv7"]
Sep 10 17:45:49.114786 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:49.114752    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56cb3fe-3eaa-4340-9a51-3cff321c98b1" path="/var/lib/kubelet/pods/c56cb3fe-3eaa-4340-9a51-3cff321c98b1/volumes"
Sep 10 17:45:49.281988 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:49.281952    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:45:53.380502 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:53.380468    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"]
Sep 10 17:45:53.380874 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:53.380726    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy" containerID="cri-o://210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d" gracePeriod=30
Sep 10 17:45:58.167430 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.167394    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:45:58.629363 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.629338    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:45:58.751043 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.750953    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-certs\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751043 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.750997    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-socket\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751253 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751050    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863398a8-9b58-4fef-a919-d1b98e397783-istio-podinfo\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751253 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751094    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863398a8-9b58-4fef-a919-d1b98e397783-istiod-ca-cert\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751253 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751186    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d44d7\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-kube-api-access-d44d7\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751253 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751225    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-credential-socket\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751253 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751249    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-envoy\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751277    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-istio-token\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751281    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:58.751573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751325    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-data\") pod \"863398a8-9b58-4fef-a919-d1b98e397783\" (UID: \"863398a8-9b58-4fef-a919-d1b98e397783\") "
Sep 10 17:45:58.751573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751324    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:58.751573 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751468    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863398a8-9b58-4fef-a919-d1b98e397783-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:45:58.751893 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751676    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.751893 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751704    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-workload-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.751893 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751717    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863398a8-9b58-4fef-a919-d1b98e397783-istiod-ca-cert\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.751893 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751670    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:58.752085 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.751958    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-data" (OuterVolumeSpecName: "istio-data") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:58.753620 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.753577    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-istio-token" (OuterVolumeSpecName: "istio-token") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:58.753732 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.753630    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-kube-api-access-d44d7" (OuterVolumeSpecName: "kube-api-access-d44d7") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "kube-api-access-d44d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:45:58.753791 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.753771    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/863398a8-9b58-4fef-a919-d1b98e397783-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 17:45:58.753866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.753850    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "863398a8-9b58-4fef-a919-d1b98e397783" (UID: "863398a8-9b58-4fef-a919-d1b98e397783"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:45:58.852174 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.852146    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-istio-token\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.852174 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.852170    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-data\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.852376 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.852183    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863398a8-9b58-4fef-a919-d1b98e397783-istio-podinfo\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.852376 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.852195    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44d7\" (UniqueName: \"kubernetes.io/projected/863398a8-9b58-4fef-a919-d1b98e397783-kube-api-access-d44d7\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.852376 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.852204    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-credential-socket\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:58.852376 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:58.852218    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863398a8-9b58-4fef-a919-d1b98e397783-istio-envoy\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:45:59.496650 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.496559    2554 generic.go:356] "Generic (PLEG): container finished" podID="863398a8-9b58-4fef-a919-d1b98e397783" containerID="210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d" exitCode=0
Sep 10 17:45:59.496650 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.496604    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" event={"ID":"863398a8-9b58-4fef-a919-d1b98e397783","Type":"ContainerDied","Data":"210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d"}
Sep 10 17:45:59.496650 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.496631    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"
Sep 10 17:45:59.497244 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.496689    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42" event={"ID":"863398a8-9b58-4fef-a919-d1b98e397783","Type":"ContainerDied","Data":"f17a70072b95ad1ab7e25a9722a044add80a9dc3ab502aa099ae41653a46b30b"}
Sep 10 17:45:59.497244 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.496712    2554 scope.go:117] "RemoveContainer" containerID="210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d"
Sep 10 17:45:59.508170 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.507983    2554 scope.go:117] "RemoveContainer" containerID="1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb"
Sep 10 17:45:59.516127 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.516109    2554 scope.go:117] "RemoveContainer" containerID="210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d"
Sep 10 17:45:59.516471 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:45:59.516445    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d\": container with ID starting with 210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d not found: ID does not exist" containerID="210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d"
Sep 10 17:45:59.516587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.516482    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d"} err="failed to get container status \"210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d\": rpc error: code = NotFound desc = could not find container \"210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d\": container with ID starting with 210ae06e64c9a2bf16b9be35a0879ed3058ba393e6226e687601f7282843235d not found: ID does not exist"
Sep 10 17:45:59.516587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.516506    2554 scope.go:117] "RemoveContainer" containerID="1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb"
Sep 10 17:45:59.516857 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:45:59.516837    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb\": container with ID starting with 1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb not found: ID does not exist" containerID="1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb"
Sep 10 17:45:59.516913 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.516863    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb"} err="failed to get container status \"1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb\": rpc error: code = NotFound desc = could not find container \"1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb\": container with ID starting with 1add14d44f1111ecd304723458e728c13d203d7dc080f6d1edc808a4c04094cb not found: ID does not exist"
Sep 10 17:45:59.517645 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.517621    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"]
Sep 10 17:45:59.521032 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:45:59.521014    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c1fb61-openshift-default-866f8856fb-5fl42"]
Sep 10 17:46:01.114012 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:46:01.113979    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863398a8-9b58-4fef-a919-d1b98e397783" path="/var/lib/kubelet/pods/863398a8-9b58-4fef-a919-d1b98e397783/volumes"
Sep 10 17:46:25.415875 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:46:25.415796    2554 scope.go:117] "RemoveContainer" containerID="0fe26bcb21597f0e188d58816cabc7751ddf72e9b2ce07837f3e1404e218b0f3"
Sep 10 17:46:25.424060 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:46:25.424042    2554 scope.go:117] "RemoveContainer" containerID="7db18157b1497de8ea8cf198d1f487e5c2a37afb4c30e5b07a3fc1f264247919"
Sep 10 17:46:25.431778 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:46:25.431754    2554 scope.go:117] "RemoveContainer" containerID="4f5a7c3a53b21cc0ae4a4ff565548339b2050cd1a9c21490cd0b86f2cdfd1ffc"
Sep 10 17:47:20.964155 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:47:20.964118    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:47:45.063558 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:47:45.063508    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:47:53.170438 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:47:53.170355    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:48:01.363491 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:48:01.363455    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:48:15.491151 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:48:15.491114    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:48:24.172347 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:48:24.172310    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:48:25.183043 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:48:25.183015    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:48:25.189517 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:48:25.189493    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:49:53.865046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:49:53.864954    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:50:02.564602 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:02.564561    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:50:10.666550 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:10.666499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:50:11.502379 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.502343    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"]
Sep 10 17:50:11.502866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.502835    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy"
Sep 10 17:50:11.502866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.502854    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy"
Sep 10 17:50:11.502866 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.502868    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f3fc3b-9194-43df-b6dd-d9d93cf32e45" containerName="revoke-keys"
Sep 10 17:50:11.503113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.502908    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f3fc3b-9194-43df-b6dd-d9d93cf32e45" containerName="revoke-keys"
Sep 10 17:50:11.503113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.502940    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56cb3fe-3eaa-4340-9a51-3cff321c98b1" containerName="payload-processing"
Sep 10 17:50:11.503113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.502949    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56cb3fe-3eaa-4340-9a51-3cff321c98b1" containerName="payload-processing"
Sep 10 17:50:11.503113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.503028    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f3fc3b-9194-43df-b6dd-d9d93cf32e45" containerName="revoke-keys"
Sep 10 17:50:11.503113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.503040    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy"
Sep 10 17:50:11.503113 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.503052    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56cb3fe-3eaa-4340-9a51-3cff321c98b1" containerName="payload-processing"
Sep 10 17:50:11.506378 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.506358    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.508936 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.508918    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-60869e21-kserve-self-signed-certs\""
Sep 10 17:50:11.515403 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.515376    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"]
Sep 10 17:50:11.625247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.620413    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tls-certs\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.625247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.620495    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-model-cache\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.625247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.620550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd9l\" (UniqueName: \"kubernetes.io/projected/4eb72b9c-52cd-41c5-bd3b-188a67656c18-kube-api-access-bmd9l\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.625247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.620580    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-dshm\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.625247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.620620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tmp-dir\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.625247 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.620653    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-home\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.721619 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.721584    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tls-certs\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.721642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-model-cache\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.721664    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd9l\" (UniqueName: \"kubernetes.io/projected/4eb72b9c-52cd-41c5-bd3b-188a67656c18-kube-api-access-bmd9l\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.721683    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-dshm\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.721705    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tmp-dir\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.721728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-home\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722281 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.722137    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-home\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722281 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.722220    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tmp-dir\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.722281 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.722240    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-model-cache\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.724009 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.723987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-dshm\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.724122 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.724102    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tls-certs\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.729671 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.729650    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd9l\" (UniqueName: \"kubernetes.io/projected/4eb72b9c-52cd-41c5-bd3b-188a67656c18-kube-api-access-bmd9l\") pod \"e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") " pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:11.818086 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:11.818056    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:12.088985 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:12.088848    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"]
Sep 10 17:50:12.091206 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:50:12.091178    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eb72b9c_52cd_41c5_bd3b_188a67656c18.slice/crio-feb000f77a990a620a2412e3b921d5d9f604f2430cc7fe4b1c455cc046d408d9 WatchSource:0}: Error finding container feb000f77a990a620a2412e3b921d5d9f604f2430cc7fe4b1c455cc046d408d9: Status 404 returned error can't find the container with id feb000f77a990a620a2412e3b921d5d9f604f2430cc7fe4b1c455cc046d408d9
Sep 10 17:50:12.699323 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:12.699285    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9" event={"ID":"4eb72b9c-52cd-41c5-bd3b-188a67656c18","Type":"ContainerStarted","Data":"8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f"}
Sep 10 17:50:12.699498 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:12.699335    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9" event={"ID":"4eb72b9c-52cd-41c5-bd3b-188a67656c18","Type":"ContainerStarted","Data":"feb000f77a990a620a2412e3b921d5d9f604f2430cc7fe4b1c455cc046d408d9"}
Sep 10 17:50:12.718080 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:12.718031    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9" podStartSLOduration=1.718017489 podStartE2EDuration="1.718017489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:12.715842426 +0000 UTC m=+2208.072332827" watchObservedRunningTime="2026-09-10 17:50:12.718017489 +0000 UTC m=+2208.074507889"
Sep 10 17:50:19.371758 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:19.371721    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:50:21.818613 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:21.818576    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:21.819049 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:21.818626    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:21.831263 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:21.831239    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:22.758911 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:22.758879    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:50:25.163403 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:25.163355    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:50:25.492363 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:25.492283    2554 scope.go:117] "RemoveContainer" containerID="93e1a68adf1dfd0aca968f5c0394e691d375e4308807d3d8ca2875dc177af6d3"
Sep 10 17:50:25.503674 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:25.503653    2554 scope.go:117] "RemoveContainer" containerID="594bcfe678508ad13db3d0cbc28735a6dcf3a3f36205d6ce8926b98e59936978"
Sep 10 17:50:56.520591 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:50:56.520497    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"]
Sep 10 17:51:00.073454 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:00.073416    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:51:09.674633 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:09.674600    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:51:11.539362 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.539302    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9" podUID="4eb72b9c-52cd-41c5-bd3b-188a67656c18" containerName="main" containerID="cri-o://8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f" gracePeriod=45
Sep 10 17:51:11.791303 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.791247    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:51:11.969452 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969413    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-model-cache\") pod \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") "
Sep 10 17:51:11.969661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969471    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tmp-dir\") pod \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") "
Sep 10 17:51:11.969661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969515    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmd9l\" (UniqueName: \"kubernetes.io/projected/4eb72b9c-52cd-41c5-bd3b-188a67656c18-kube-api-access-bmd9l\") pod \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") "
Sep 10 17:51:11.969661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969586    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tls-certs\") pod \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") "
Sep 10 17:51:11.969661 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969631    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-home\") pod \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") "
Sep 10 17:51:11.969907 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969673    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-dshm\") pod \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\" (UID: \"4eb72b9c-52cd-41c5-bd3b-188a67656c18\") "
Sep 10 17:51:11.969907 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969796    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-model-cache" (OuterVolumeSpecName: "model-cache") pod "4eb72b9c-52cd-41c5-bd3b-188a67656c18" (UID: "4eb72b9c-52cd-41c5-bd3b-188a67656c18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:11.969907 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969817    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4eb72b9c-52cd-41c5-bd3b-188a67656c18" (UID: "4eb72b9c-52cd-41c5-bd3b-188a67656c18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:11.969907 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.969879    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-home" (OuterVolumeSpecName: "home") pod "4eb72b9c-52cd-41c5-bd3b-188a67656c18" (UID: "4eb72b9c-52cd-41c5-bd3b-188a67656c18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:11.970127 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.970033    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-model-cache\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:51:11.970127 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.970048    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tmp-dir\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:51:11.970127 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.970056    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-home\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:51:11.971760 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.971731    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4eb72b9c-52cd-41c5-bd3b-188a67656c18" (UID: "4eb72b9c-52cd-41c5-bd3b-188a67656c18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:51:11.971871 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.971827    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-dshm" (OuterVolumeSpecName: "dshm") pod "4eb72b9c-52cd-41c5-bd3b-188a67656c18" (UID: "4eb72b9c-52cd-41c5-bd3b-188a67656c18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:51:11.971942 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.971921    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb72b9c-52cd-41c5-bd3b-188a67656c18-kube-api-access-bmd9l" (OuterVolumeSpecName: "kube-api-access-bmd9l") pod "4eb72b9c-52cd-41c5-bd3b-188a67656c18" (UID: "4eb72b9c-52cd-41c5-bd3b-188a67656c18"). InnerVolumeSpecName "kube-api-access-bmd9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:51:11.990633 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.990599    2554 generic.go:356] "Generic (PLEG): container finished" podID="4eb72b9c-52cd-41c5-bd3b-188a67656c18" containerID="8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f" exitCode=0
Sep 10 17:51:11.990753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.990643    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"
Sep 10 17:51:11.990753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.990682    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9" event={"ID":"4eb72b9c-52cd-41c5-bd3b-188a67656c18","Type":"ContainerDied","Data":"8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f"}
Sep 10 17:51:11.990753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.990727    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9" event={"ID":"4eb72b9c-52cd-41c5-bd3b-188a67656c18","Type":"ContainerDied","Data":"feb000f77a990a620a2412e3b921d5d9f604f2430cc7fe4b1c455cc046d408d9"}
Sep 10 17:51:11.990753 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.990741    2554 scope.go:117] "RemoveContainer" containerID="8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f"
Sep 10 17:51:11.999676 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.999657    2554 scope.go:117] "RemoveContainer" containerID="8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f"
Sep 10 17:51:11.999954 ip-10-0-142-52 kubenswrapper[2554]: E0910 17:51:11.999929    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f\": container with ID starting with 8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f not found: ID does not exist" containerID="8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f"
Sep 10 17:51:12.000046 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:11.999962    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f"} err="failed to get container status \"8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f\": rpc error: code = NotFound desc = could not find container \"8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f\": container with ID starting with 8a9200c4359758bef6c66958f5049d31edd0be4b6216183fd1baa8970b4e1a9f not found: ID does not exist"
Sep 10 17:51:12.011934 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:12.011907    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"]
Sep 10 17:51:12.019854 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:12.019830    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-60869e21-kserve-6897fbd88c-4hrh9"]
Sep 10 17:51:12.071278 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:12.071257    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmd9l\" (UniqueName: \"kubernetes.io/projected/4eb72b9c-52cd-41c5-bd3b-188a67656c18-kube-api-access-bmd9l\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:51:12.071278 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:12.071277    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb72b9c-52cd-41c5-bd3b-188a67656c18-tls-certs\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:51:12.071413 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:12.071289    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb72b9c-52cd-41c5-bd3b-188a67656c18-dshm\") on node \"ip-10-0-142-52.ec2.internal\" DevicePath \"\""
Sep 10 17:51:13.114816 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:13.114770    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb72b9c-52cd-41c5-bd3b-188a67656c18" path="/var/lib/kubelet/pods/4eb72b9c-52cd-41c5-bd3b-188a67656c18/volumes"
Sep 10 17:51:16.167463 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:16.167419    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:51:24.367730 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:24.367694    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:51:25.526278 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:25.526242    2554 scope.go:117] "RemoveContainer" containerID="f1873ef7dc620b0770934b354b26ddd354b7ac4ea683d9fc6150a1d159b9be40"
Sep 10 17:51:25.534602 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:25.534568    2554 scope.go:117] "RemoveContainer" containerID="10489d5815b22cfd1400620967a5301ec935aaa5c23e2fcbc9538acabc5b6fc6"
Sep 10 17:51:25.543062 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:25.543034    2554 scope.go:117] "RemoveContainer" containerID="8622611591b8cdb07397896cee12de9b01df979b3502e13f2d74f24d4379879c"
Sep 10 17:51:25.551342 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:25.551318    2554 scope.go:117] "RemoveContainer" containerID="a89e310b9322f43d0969b9a37349a414fcfd8811168def3823e8a56195608575"
Sep 10 17:51:30.467346 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:30.467310    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:51:39.968851 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:39.968815    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:51:46.268256 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:46.268221    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-pwczx"]
Sep 10 17:51:49.611362 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:49.611332    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775ccfbcbd-w4z5r_c4789de8-e863-4760-8ecd-06118e8b484f/authorino/0.log"
Sep 10 17:51:56.082330 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:56.082293    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-vn8sj_632520e6-0e12-4727-ba0f-2ce2a2d3ca20/copy-manifests/0.log"
Sep 10 17:51:56.089341 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:56.089319    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-vn8sj_632520e6-0e12-4727-ba0f-2ce2a2d3ca20/manager/0.log"
Sep 10 17:51:56.660556 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:56.660517    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817690-8sdc5_7877d0a2-acf9-476a-a048-b54896f5f870/cleanup/0.log"
Sep 10 17:51:57.689679 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:57.689627    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775ccfbcbd-w4z5r_c4789de8-e863-4760-8ecd-06118e8b484f/authorino/0.log"
Sep 10 17:51:58.039353 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:58.039264    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-x8jmm_d37b5b84-9568-44ec-9acd-22f1257d36ef/kuadrant-console-plugin/0.log"
Sep 10 17:51:58.293039 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:58.292952    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-pwczx_9bdef861-edc3-4c74-bc0a-eb9056e647c7/limitador/0.log"
Sep 10 17:51:58.994641 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:51:58.994606    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-jkz7n_f6d74e93-a104-4a56-a97f-716edd7b1db5/kube-auth-proxy/0.log"
Sep 10 17:52:00.207516 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:00.207481    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-4kltf_0f74d224-0543-4664-8236-350bd39b007c/main/0.log"
Sep 10 17:52:00.437360 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:00.437323    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7phlv_e4c70b29-e439-4b97-947f-32f510cb8f8d/main/0.log"
Sep 10 17:52:00.562422 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:00.562394    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-pwns8_81c0372c-0cf3-4f2c-a725-1ffb7b83681e/main/0.log"
Sep 10 17:52:00.680690 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:00.680659    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2tx6s_aaf3fb73-aa5f-4f97-81d4-fdefa589e9af/main/0.log"
Sep 10 17:52:09.432811 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:09.432776    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-499fh_4ff0d9fd-0612-4d96-b385-235ce5bcbed0/global-pull-secret-syncer/0.log"
Sep 10 17:52:09.652329 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:09.652295    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qxtsv_75aa5079-794a-466b-ba77-8a88ce296e52/konnectivity-agent/0.log"
Sep 10 17:52:09.722627 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:09.722550    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-52.ec2.internal_8f119684184aeb2e23bdcf787e218d99/haproxy/0.log"
Sep 10 17:52:15.099932 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:15.099894    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-775ccfbcbd-w4z5r_c4789de8-e863-4760-8ecd-06118e8b484f/authorino/0.log"
Sep 10 17:52:15.206692 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:15.206659    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-x8jmm_d37b5b84-9568-44ec-9acd-22f1257d36ef/kuadrant-console-plugin/0.log"
Sep 10 17:52:15.288085 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:15.288052    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-pwczx_9bdef861-edc3-4c74-bc0a-eb9056e647c7/limitador/0.log"
Sep 10 17:52:17.321460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.321426    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6988d857c9-dfgng_f6827ab9-4080-4d45-841b-11630b5e25ba/metrics-server/0.log"
Sep 10 17:52:17.384501 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.384467    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9c2g9_f5bc03b2-3f00-4aa7-b90c-82a09fb0e110/node-exporter/0.log"
Sep 10 17:52:17.404050 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.404024    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9c2g9_f5bc03b2-3f00-4aa7-b90c-82a09fb0e110/kube-rbac-proxy/0.log"
Sep 10 17:52:17.436460 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.436438    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9c2g9_f5bc03b2-3f00-4aa7-b90c-82a09fb0e110/init-textfile/0.log"
Sep 10 17:52:17.746036 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.746003    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_100f559f-2a2a-4980-adcc-4c54bad7b39c/prometheus/0.log"
Sep 10 17:52:17.766465 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.766442    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_100f559f-2a2a-4980-adcc-4c54bad7b39c/config-reloader/0.log"
Sep 10 17:52:17.789451 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.789429    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_100f559f-2a2a-4980-adcc-4c54bad7b39c/thanos-sidecar/0.log"
Sep 10 17:52:17.812496 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.812436    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_100f559f-2a2a-4980-adcc-4c54bad7b39c/kube-rbac-proxy-web/0.log"
Sep 10 17:52:17.833014 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.832992    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_100f559f-2a2a-4980-adcc-4c54bad7b39c/kube-rbac-proxy/0.log"
Sep 10 17:52:17.854298 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.854279    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_100f559f-2a2a-4980-adcc-4c54bad7b39c/kube-rbac-proxy-thanos/0.log"
Sep 10 17:52:17.876364 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:17.876344    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_100f559f-2a2a-4980-adcc-4c54bad7b39c/init-config-reloader/0.log"
Sep 10 17:52:20.030232 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.030164    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/2.log"
Sep 10 17:52:20.034471 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.034450    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z7qwj_b786b74c-bc04-4817-b490-c2d209a18cdb/console-operator/3.log"
Sep 10 17:52:20.153220 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.153190    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"]
Sep 10 17:52:20.153515 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.153503    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb72b9c-52cd-41c5-bd3b-188a67656c18" containerName="main"
Sep 10 17:52:20.153587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.153517    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb72b9c-52cd-41c5-bd3b-188a67656c18" containerName="main"
Sep 10 17:52:20.153587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.153554    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy"
Sep 10 17:52:20.153587 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.153560    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy"
Sep 10 17:52:20.153682 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.153609    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb72b9c-52cd-41c5-bd3b-188a67656c18" containerName="main"
Sep 10 17:52:20.153682 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.153617    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="863398a8-9b58-4fef-a919-d1b98e397783" containerName="istio-proxy"
Sep 10 17:52:20.156590 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.156570    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.159372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.159353    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7ws5r\"/\"openshift-service-ca.crt\""
Sep 10 17:52:20.159495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.159440    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7ws5r\"/\"kube-root-ca.crt\""
Sep 10 17:52:20.160364 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.160348    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7ws5r\"/\"default-dockercfg-9c22d\""
Sep 10 17:52:20.166667 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.166647    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"]
Sep 10 17:52:20.167958 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.167939    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-proc\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.168045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.167972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-sys\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.168045 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.168003    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9tg7\" (UniqueName: \"kubernetes.io/projected/1847d145-a639-4a46-9fea-a4e4a883bcfb-kube-api-access-s9tg7\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.168135 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.168111    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-podres\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.168233 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.168211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-lib-modules\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269033 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.268998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-lib-modules\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-proc\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269071    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-sys\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269111    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9tg7\" (UniqueName: \"kubernetes.io/projected/1847d145-a639-4a46-9fea-a4e4a883bcfb-kube-api-access-s9tg7\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269137    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-proc\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269150    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-podres\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269206 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269180    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-lib-modules\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269216    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-sys\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.269419 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.269245    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1847d145-a639-4a46-9fea-a4e4a883bcfb-podres\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.281453 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.281390    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9tg7\" (UniqueName: \"kubernetes.io/projected/1847d145-a639-4a46-9fea-a4e4a883bcfb-kube-api-access-s9tg7\") pod \"perf-node-gather-daemonset-4n8f9\" (UID: \"1847d145-a639-4a46-9fea-a4e4a883bcfb\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.466765 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.466729    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:20.742775 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:20.742749    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"]
Sep 10 17:52:20.745282 ip-10-0-142-52 kubenswrapper[2554]: W0910 17:52:20.745251    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1847d145_a639_4a46_9fea_a4e4a883bcfb.slice/crio-a4ecc663e4a46eeaaa33ada49d227de46eb6c93015ae20ff33d79ebb84df11b1 WatchSource:0}: Error finding container a4ecc663e4a46eeaaa33ada49d227de46eb6c93015ae20ff33d79ebb84df11b1: Status 404 returned error can't find the container with id a4ecc663e4a46eeaaa33ada49d227de46eb6c93015ae20ff33d79ebb84df11b1
Sep 10 17:52:21.300372 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:21.300332    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9" event={"ID":"1847d145-a639-4a46-9fea-a4e4a883bcfb","Type":"ContainerStarted","Data":"61d2372c4bb0e3589a91986dec007dfecb799b5d0e5cf95485f63e11f31bde7f"}
Sep 10 17:52:21.300787 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:21.300389    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9" event={"ID":"1847d145-a639-4a46-9fea-a4e4a883bcfb","Type":"ContainerStarted","Data":"a4ecc663e4a46eeaaa33ada49d227de46eb6c93015ae20ff33d79ebb84df11b1"}
Sep 10 17:52:21.300787 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:21.300443    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:21.320135 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:21.320085    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9" podStartSLOduration=1.320071939 podStartE2EDuration="1.320071939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:52:21.317655783 +0000 UTC m=+2336.674146184" watchObservedRunningTime="2026-09-10 17:52:21.320071939 +0000 UTC m=+2336.676562339"
Sep 10 17:52:21.338433 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:21.338412    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-xg8cj_6f89f9bc-5619-49ae-9f52-3409196a37f5/volume-data-source-validator/0.log"
Sep 10 17:52:22.362924 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:22.362893    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8m29g_098a0f94-e1af-4846-b58b-d4e132ae8d7b/dns/0.log"
Sep 10 17:52:22.386126 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:22.386094    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8m29g_098a0f94-e1af-4846-b58b-d4e132ae8d7b/kube-rbac-proxy/0.log"
Sep 10 17:52:22.543473 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:22.543436    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ss6gj_22d0f763-d51f-4444-89fd-0732b083afa3/dns-node-resolver/0.log"
Sep 10 17:52:23.095495 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:23.095468    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-9bbd76d7f-v7d7k_97e1b1c5-8766-4bb5-b4df-7e9a7f5220fb/registry/0.log"
Sep 10 17:52:23.163065 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:23.163034    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zpxkk_e24a6e41-b03d-4e48-be14-b7487072c80b/node-ca/0.log"
Sep 10 17:52:24.523156 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:24.523130    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-jkz7n_f6d74e93-a104-4a56-a97f-716edd7b1db5/kube-auth-proxy/0.log"
Sep 10 17:52:25.558080 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:25.558050    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7mrft_b85b6068-2505-4ed6-9dfa-7e5f9841a3a9/serve-healthcheck-canary/0.log"
Sep 10 17:52:25.586653 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:25.586627    2554 scope.go:117] "RemoveContainer" containerID="9a7dbbc9fe55b97a606a5ff3dcce025a40d35e68809f46090126423b0b253b1b"
Sep 10 17:52:26.196648 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:26.196618    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-4dk2c_b30b0823-6551-4431-91ef-bccae87c4810/insights-operator/0.log"
Sep 10 17:52:26.197244 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:26.197225    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-4dk2c_b30b0823-6551-4431-91ef-bccae87c4810/insights-operator/1.log"
Sep 10 17:52:26.364652 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:26.364595    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cz8pq_f4f8197d-f3d1-4bc5-8e68-bb82c19b972b/kube-rbac-proxy/0.log"
Sep 10 17:52:26.386075 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:26.386053    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cz8pq_f4f8197d-f3d1-4bc5-8e68-bb82c19b972b/exporter/0.log"
Sep 10 17:52:26.406069 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:26.406049    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cz8pq_f4f8197d-f3d1-4bc5-8e68-bb82c19b972b/extractor/0.log"
Sep 10 17:52:27.314626 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:27.314601    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-4n8f9"
Sep 10 17:52:29.150569 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:29.150524    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-vn8sj_632520e6-0e12-4727-ba0f-2ce2a2d3ca20/manager/0.log"
Sep 10 17:52:29.171595 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:29.171564    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-vn8sj_632520e6-0e12-4727-ba0f-2ce2a2d3ca20/copy-manifests/0.log"
Sep 10 17:52:30.499248 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:30.499216    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8655464fff-62rkw_6eaea14d-952f-454e-b5fd-d411d549cb1c/manager/0.log"
Sep 10 17:52:31.519980 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:31.519952    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-jkz7n_f6d74e93-a104-4a56-a97f-716edd7b1db5/kube-auth-proxy/0.log"
Sep 10 17:52:33.461422 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:33.461389    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-jkz7n_f6d74e93-a104-4a56-a97f-716edd7b1db5/kube-auth-proxy/0.log"
Sep 10 17:52:37.370153 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:37.370119    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvwnv_873150c3-9650-4c7e-b4ba-5f44b32f2127/kube-multus-additional-cni-plugins/0.log"
Sep 10 17:52:37.408123 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:37.408091    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvwnv_873150c3-9650-4c7e-b4ba-5f44b32f2127/egress-router-binary-copy/0.log"
Sep 10 17:52:37.444182 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:37.444159    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvwnv_873150c3-9650-4c7e-b4ba-5f44b32f2127/cni-plugins/0.log"
Sep 10 17:52:37.481966 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:37.481905    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvwnv_873150c3-9650-4c7e-b4ba-5f44b32f2127/bond-cni-plugin/0.log"
Sep 10 17:52:37.521083 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:37.521062    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvwnv_873150c3-9650-4c7e-b4ba-5f44b32f2127/routeoverride-cni/0.log"
Sep 10 17:52:37.553044 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:37.553025    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvwnv_873150c3-9650-4c7e-b4ba-5f44b32f2127/whereabouts-cni-bincopy/0.log"
Sep 10 17:52:37.585081 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:37.585061    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dvwnv_873150c3-9650-4c7e-b4ba-5f44b32f2127/whereabouts-cni/0.log"
Sep 10 17:52:38.276213 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:38.276187    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xq984_1ec44f3f-d2d1-4e65-bd8b-8af8a81951ad/kube-multus/0.log"
Sep 10 17:52:38.314233 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:38.314197    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n7w82_d1cbbfd7-c7ab-4420-8376-3cbd71ed8829/network-metrics-daemon/0.log"
Sep 10 17:52:38.347519 ip-10-0-142-52 kubenswrapper[2554]: I0910 17:52:38.347494    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n7w82_d1cbbfd7-c7ab-4420-8376-3cbd71ed8829/kube-rbac-proxy/0.log"