Sep 10 14:45:45.245011 ip-10-0-130-2 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 14:45:45.245021 ip-10-0-130-2 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 14:45:45.245028 ip-10-0-130-2 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 14:45:45.245168 ip-10-0-130-2 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 14:45:55.252533 ip-10-0-130-2 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 14:45:55.252549 ip-10-0-130-2 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bb1b0a8b3963415f817d8ff344ec3f86 --
Sep 10 14:47:43.769409 ip-10-0-130-2 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 14:47:44.196283 ip-10-0-130-2 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:47:44.196283 ip-10-0-130-2 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 14:47:44.196283 ip-10-0-130-2 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:47:44.196283 ip-10-0-130-2 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 14:47:44.196283 ip-10-0-130-2 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:47:44.197402 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.197321    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 14:47:44.201393 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201378    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:47:44.201393 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201394    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201398    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201401    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201404    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201407    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201410    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201412    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201415    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201418    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201421    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201424    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201427    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201430    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201432    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201435    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201438    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201441    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201444    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201446    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201449    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:47:44.201459 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201452    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201455    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201457    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201460    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201463    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201466    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201469    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201472    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201475    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201478    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201481    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201483    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201486    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201488    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201493    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201496    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201499    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201502    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201504    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:47:44.201939 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201507    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201510    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201512    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201515    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201517    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201520    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201522    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201525    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201529    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201533    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201536    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201538    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201541    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201544    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201546    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201550    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201553    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201556    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201559    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:47:44.202453 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201562    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201564    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201567    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201569    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201572    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201574    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201577    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201580    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201582    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201585    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201588    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201590    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201593    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201597    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201600    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201602    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201604    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201607    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201610    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201613    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:47:44.202916 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201616    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201619    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201622    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201624    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201627    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201630    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201634    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201638    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201641    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201644    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201647    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201650    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201653    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201655    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201658    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201661    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201664    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201666    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201669    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.201671    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:47:44.203437 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202371    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202378    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202381    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202384    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202387    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202390    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202393    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202396    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202398    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202401    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202403    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202406    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202410    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202413    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202416    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202419    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202421    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202424    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202427    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202429    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:47:44.203934 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202432    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202437    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202440    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202443    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202446    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202449    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202451    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202454    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202457    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202460    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202463    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202465    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202468    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202471    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202474    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202477    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202479    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202482    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202485    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202490    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:47:44.204452 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202494    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202497    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202500    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202503    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202506    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202509    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202512    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202516    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202519    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202522    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202525    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202527    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202531    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202534    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202537    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202539    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202542    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202544    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202547    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:47:44.204938 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202550    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202552    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202555    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202557    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202560    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202563    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202565    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202567    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202571    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202574    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202576    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202579    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202582    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202584    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202587    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202589    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202593    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202596    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202598    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:47:44.205416 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202601    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202603    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202606    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202608    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202611    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202613    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202616    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202619    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202622    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202624    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202627    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202630    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202632    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202635    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202637    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202640    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202642    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202645    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202647    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:47:44.205886 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202650    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.202652    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203032    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203037    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203040    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203044    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203047    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203050    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203053    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203057    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203060    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203063    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203066    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203068    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203071    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203073    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203076    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203079    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203081    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203084    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:47:44.206392 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203087    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203090    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203092    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203095    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203099    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203103    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203106    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203108    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203111    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203113    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203116    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203118    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203136    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203140    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203143    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203146    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203148    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203151    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203154    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:47:44.206877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203157    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203159    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203163    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203166    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203168    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203171    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203174    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203176    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203179    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203182    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203184    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203188    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203190    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203193    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203195    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203198    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203200    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203203    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203206    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:47:44.207358 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203208    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203211    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203213    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203216    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203219    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203221    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203224    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203226    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203229    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203231    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203234    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203236    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203239    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203241    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203246    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203253    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203256    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203259    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203262    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203265    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:47:44.207846 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203268    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203271    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203275    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203278    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203281    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203284    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203287    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203290    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203292    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203295    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203298    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203301    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203304    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203307    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203310    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203312    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203314    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203317    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203320    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:47:44.208361 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203322    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203325    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203327    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.203330    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203402    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203437    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203445    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203451    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203458    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203461    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203465    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203470    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203473    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203475    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203479    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203482    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203485    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203488    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203491    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203494    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203497    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203500    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203504    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203507    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 14:47:44.208852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203510    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203514    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203517    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203521    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203524    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203527    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203530    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203534    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203537    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203540    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203543    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203545    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203550    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203553    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203555    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203558    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203562    2565 flags.go:64] FLAG: --enable-server="true"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203566    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203594    2565 flags.go:64] FLAG: --event-burst="100"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203598    2565 flags.go:64] FLAG: --event-qps="50"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203601    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203605    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203608    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203612    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 14:47:44.209444 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203614    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203617    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203620    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203623    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203626    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203629    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203632    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203635    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203638    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203640    2565 flags.go:64] FLAG: --feature-gates=""
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203644    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203647    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203650    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203653    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203656    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203659    2565 flags.go:64] FLAG: --help="false"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203661    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203666    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203669    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203672    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203675    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203678    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203681    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 14:47:44.210021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203684    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203688    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203691    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203699    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203702    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203705    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203708    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203711    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203714    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203717    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203720    2565 flags.go:64] FLAG: --lock-file=""
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203723    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203726    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203729    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203734    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203737    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203740    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203742    2565 flags.go:64] FLAG: --logging-format="text"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203745    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203748    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203751    2565 flags.go:64] FLAG: --manifest-url=""
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203754    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203758    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203762    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203766    2565 flags.go:64] FLAG: --max-pods="110"
Sep 10 14:47:44.210643 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203768    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203773    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203776    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203779    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203782    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203784    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203788    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203794    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203798    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203801    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203804    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203807    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203812    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203815    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203819    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203821    2565 flags.go:64] FLAG: --port="10250"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203824    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203827    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038d9fbe4d02b1cbe"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203831    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203834    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203837    2565 flags.go:64] FLAG: --register-node="true"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203839    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203843    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203846    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 10 14:47:44.211264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203848    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203851    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203855    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203858    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203861    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203864    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203867    2565 flags.go:64] FLAG: --runonce="false"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203870    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203872    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203877    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203880    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203882    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203885    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203888    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203891    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203894    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203897    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203901    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203904    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203907    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203910    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203913    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203918    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203921    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203924    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 14:47:44.211846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203928    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203930    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203933    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203936    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203939    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203942    2565 flags.go:64] FLAG: --v="2"
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203946    2565 flags.go:64] FLAG: --version="false"
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203950    2565 flags.go:64] FLAG: --vmodule=""
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203962    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.203966    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204097    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204101    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204105    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204108    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204111    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204113    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204118    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204132    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204135    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204138    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204140    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204143    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:47:44.212514 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204146    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204148    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204152    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204154    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204157    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204160    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204163    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204166    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204168    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204171    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204174    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204177    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204179    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204182    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204184    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204187    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204189    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204192    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204195    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204197    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:47:44.213156 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204200    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204202    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204204    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204207    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204209    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204212    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204218    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204221    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204224    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204227    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204230    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204233    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204235    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204238    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204241    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204244    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204247    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204250    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204253    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:47:44.213668 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204256    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204259    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204261    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204263    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204266    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204269    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204271    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204273    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204276    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204279    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204281    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204284    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204287    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204289    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204292    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204295    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204297    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204300    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204303    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204308    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:47:44.214189 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204311    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204313    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204316    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204319    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204321    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204324    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204326    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204329    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204332    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204334    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204337    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204339    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204342    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204345    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204347    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204350    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204352    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204355    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204357    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204360    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:47:44.214687 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204362    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204365    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204368    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204371    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204374    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204377    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204381    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.204384    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.204389    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.212301    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.212315    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.213258    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.213269    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213335    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:47:44.215184 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213340    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213344    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213346    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213349    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213353    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213356    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213359    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213362    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213366    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213370    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213373    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213382    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213386    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213390    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213394    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213396    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213399    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213402    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213405    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:47:44.215547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213408    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213410    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213413    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213415    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213418    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213420    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213423    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213426    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213428    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213432    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213434    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213437    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213440    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213442    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213446    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213448    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213451    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213453    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213456    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213458    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:47:44.216055 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213461    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213463    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213466    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213468    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213471    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213479    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213482    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213484    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213487    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213489    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213492    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213494    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213497    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213499    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213502    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213505    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213507    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213510    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213512    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213515    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:47:44.216557 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213517    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213521    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213524    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213527    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213529    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213532    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213534    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213537    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213539    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213542    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213545    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213547    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213550    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213552    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213555    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213557    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213559    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213562    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213574    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213577    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:47:44.217052 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213580    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213582    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213585    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213587    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213589    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213592    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213595    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213597    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213599    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213602    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213604    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213606    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213609    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213612    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213615    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213618    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213620    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213623    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:47:44.217571 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213625    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.213630    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213782    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213788    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213792    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213795    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213798    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213801    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213804    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213808    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213810    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213813    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213816    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213825    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:47:44.218007 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213828    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213830    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213833    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213835    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213838    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213842    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213845    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213848    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213851    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213854    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213857    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213861    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213864    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213868    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213871    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213874    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213876    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213879    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:47:44.218419 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213882    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213884    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213886    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213889    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213892    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213894    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213897    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213899    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213902    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213904    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213907    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213909    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213912    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213914    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213922    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213925    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213928    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213930    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213933    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213935    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:47:44.218853 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213938    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213940    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213943    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213945    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213947    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213950    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213952    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213955    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213957    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213960    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213962    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213965    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213968    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213970    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213973    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213975    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213978    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213980    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213983    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213985    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:47:44.219343 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213987    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213990    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213992    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213995    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213997    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.213999    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214002    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214012    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214015    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214018    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214021    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214023    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214025    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214028    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214030    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214033    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214035    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214038    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214040    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214043    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:47:44.219825 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214045    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214048    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214050    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214052    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214055    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214057    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214060    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214062    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:44.214065    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.214070    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.214209    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.217547    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.218689    2565 server.go:1019] "Starting client certificate rotation"
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.218803    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:47:44.220331 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.218842    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:47:44.221812 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.221798    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:47:44.228610 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.228511    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:47:44.239417 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.239399    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 10 14:47:44.242516 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.242492    2565 log.go:25] "Validated CRI v1 image API"
Sep 10 14:47:44.243334 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.243320    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 14:47:44.245896 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.245868    2565 fs.go:135] Filesystem UUIDs: map[164d2b66-0a24-40a8-888f-61a09bbee983:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9eb7714b-2ce9-44f8-86c2-abe69d801c63:/dev/nvme0n1p3]
Sep 10 14:47:44.245896 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.245893    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 14:47:44.250550 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.250527    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:47:44.251037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.250932    2565 manager.go:216] Machine: {Timestamp:2026-09-10 14:47:44.249513197 +0000 UTC m=+0.370569157 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100680 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210fc73ad0b4db95971b4e54db9ba3 SystemUUID:ec210fc7-3ad0-b4db-9597-1b4e54db9ba3 BootID:bb1b0a8b-3963-415f-817d-8ff344ec3f86 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a6:d8:b3:4d:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a6:d8:b3:4d:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:a7:7a:2f:ff:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 14:47:44.251037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.251037    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 14:47:44.251149 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.251117    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 14:47:44.254634 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.254611    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 14:47:44.254778 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.254637    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 14:47:44.255505 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.255495    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 14:47:44.255538 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.255508    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 14:47:44.255538 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.255516    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:47:44.256363 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.256352    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:47:44.257622 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.257612    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 14:47:44.262329 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262317    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:47:44.262436 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262427    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 14:47:44.262473 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262469    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 14:47:44.262501 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262482    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 14:47:44.262501 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262493    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 14:47:44.262501 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262502    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 10 14:47:44.262618 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262511    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 14:47:44.262954 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262943    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:47:44.262990 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.262961    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:47:44.265513 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.265495    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 14:47:44.268461 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.268446    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 14:47:44.268522 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.268478    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 14:47:44.269068 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.269054    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w4fgs"
Sep 10 14:47:44.270001 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.269988    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 14:47:44.270053 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270005    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 14:47:44.270053 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270012    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 14:47:44.270053 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270021    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 14:47:44.270053 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270033    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 14:47:44.270053 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270039    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 14:47:44.270053 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270053    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 14:47:44.270053 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270058    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 14:47:44.270274 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270064    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 14:47:44.270274 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270083    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 14:47:44.270274 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270097    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 14:47:44.270274 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.270102    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 14:47:44.271084 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.271069    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 14:47:44.271084 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.271079    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 14:47:44.273666 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.273619    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 14:47:44.273741 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.273682    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 14:47:44.273741 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.273705    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 14:47:44.275495 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.275483    2565 server.go:1268] "Started kubelet"
Sep 10 14:47:44.275593 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.275554    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 14:47:44.275795 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.275676    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 14:47:44.275877 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.275863    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 14:47:44.281255 ip-10-0-130-2 systemd[1]: Started Kubernetes Kubelet.
Sep 10 14:47:44.281521 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.280914    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 14:47:44.282446 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.281598    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w4fgs"
Sep 10 14:47:44.283328 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.283050    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 14:47:44.283610 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.283586    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 14:47:44.288417 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.287274    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-2.ec2.internal.18d3fd0285f5c43f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-2.ec2.internal,UID:ip-10-0-130-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-2.ec2.internal,},FirstTimestamp:2026-09-10 14:47:44.275457087 +0000 UTC m=+0.396513047,LastTimestamp:2026-09-10 14:47:44.275457087 +0000 UTC m=+0.396513047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-2.ec2.internal,}"
Sep 10 14:47:44.289385 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.289367    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 14:47:44.290098 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290081    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 14:47:44.290212 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.290174    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.290849 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290825    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 14:47:44.290849 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290826    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 14:47:44.290849 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290863    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 14:47:44.291012 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290938    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 14:47:44.291012 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290946    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 14:47:44.291012 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290950    2565 factory.go:55] Registering systemd factory
Sep 10 14:47:44.291012 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.290963    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 10 14:47:44.291195 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.291182    2565 factory.go:153] Registering CRI-O factory
Sep 10 14:47:44.291233 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.291200    2565 factory.go:223] Registration of the crio container factory successfully
Sep 10 14:47:44.291270 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.291249    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 14:47:44.291270 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.291265    2565 factory.go:103] Registering Raw factory
Sep 10 14:47:44.291345 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.291284    2565 manager.go:1196] Started watching for new ooms in manager
Sep 10 14:47:44.291736 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.291723    2565 manager.go:349] Starting recovery of all containers
Sep 10 14:47:44.292039 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.292019    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 14:47:44.292156 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.292116    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 14:47:44.303328 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.303312    2565 manager.go:354] Recovery completed
Sep 10 14:47:44.307975 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.307857    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:47:44.309702 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.309687    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:47:44.309784 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.309714    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:47:44.309784 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.309728    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:47:44.311149 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.311134    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 14:47:44.311149 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.311149    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 14:47:44.311277 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.311153    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:47:44.311277 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.311163    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:47:44.311832 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.311810    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:47:44.311910 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.311844    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:47:44.311910 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.311857    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:47:44.313220 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.313205    2565 policy_none.go:49] "None policy: Start"
Sep 10 14:47:44.313286 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.313224    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 14:47:44.313286 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.313238    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 14:47:44.315204 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.315193    2565 policy_none.go:47] "Start"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.352736    2565 manager.go:337] "Starting Device Plugin manager"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.352797    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.352810    2565 server.go:87] "Starting device plugin registration server"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.353141    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.353153    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.353288    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.353367    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.353376    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.353985    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 14:47:44.367639 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.354042    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.417451 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.417407    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 14:47:44.418670 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.418651    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 14:47:44.418758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.418700    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 14:47:44.418758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.418726    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 14:47:44.418844 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.418826    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 14:47:44.421590 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.421568    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:47:44.453836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.453731    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:47:44.454797 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.454779    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:47:44.454872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.454815    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:47:44.454872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.454831    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:47:44.454872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.454859    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.461592 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.461574    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.461674 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.461600    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-2.ec2.internal\": node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.483966 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.483932    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.520522 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.520488    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"]
Sep 10 14:47:44.520682 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.520560    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:47:44.521562 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.521543    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:47:44.521658 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.521576    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:47:44.521658 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.521589    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:47:44.524046 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524030    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:47:44.524164 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524147    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.524221 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524192    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:47:44.524846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524830    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:47:44.524940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524851    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:47:44.524940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524867    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:47:44.524940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524875    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:47:44.524940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524894    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:47:44.524940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.524908    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:47:44.527236 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.527221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.527311 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.527248    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:47:44.527875 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.527860    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:47:44.527940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.527879    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:47:44.527940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.527889    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:47:44.545761 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.545743    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-2.ec2.internal\" not found" node="ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.550238 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.550221    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-2.ec2.internal\" not found" node="ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.584981 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.584947    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.592372 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.592355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf2a6a1bfd0dc2f4c5a2b0190766ed9a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-2.ec2.internal\" (UID: \"bf2a6a1bfd0dc2f4c5a2b0190766ed9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.592438 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.592378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e43f56a932ead032b888e9a94f11feab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal\" (UID: \"e43f56a932ead032b888e9a94f11feab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.592438 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.592396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e43f56a932ead032b888e9a94f11feab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal\" (UID: \"e43f56a932ead032b888e9a94f11feab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.685344 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.685312    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.692679 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.692660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e43f56a932ead032b888e9a94f11feab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal\" (UID: \"e43f56a932ead032b888e9a94f11feab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.692729 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.692684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e43f56a932ead032b888e9a94f11feab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal\" (UID: \"e43f56a932ead032b888e9a94f11feab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.692729 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.692700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf2a6a1bfd0dc2f4c5a2b0190766ed9a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-2.ec2.internal\" (UID: \"bf2a6a1bfd0dc2f4c5a2b0190766ed9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.692790 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.692776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e43f56a932ead032b888e9a94f11feab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal\" (UID: \"e43f56a932ead032b888e9a94f11feab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.692829 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.692784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e43f56a932ead032b888e9a94f11feab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal\" (UID: \"e43f56a932ead032b888e9a94f11feab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.692829 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.692779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf2a6a1bfd0dc2f4c5a2b0190766ed9a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-2.ec2.internal\" (UID: \"bf2a6a1bfd0dc2f4c5a2b0190766ed9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.785911 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.785855    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.848356 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.848334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.854365 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:44.854343    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:44.886240 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.886217    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:44.986787 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:44.986749    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:46.638764 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:46.638741    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:46.739249 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:46.739227    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:46.769668 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:46.769635    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 14:47:46.770018 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:46.769766    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:47:46.807882 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:46.807862    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:47:46.839442 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:46.839421    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:46.843697 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:46.843670    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:42:45 +0000 UTC" deadline="2028-03-03 10:42:05.574247488 +0000 UTC"
Sep 10 14:47:46.843697 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:46.843695    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12955h54m18.730555222s"
Sep 10 14:47:46.940151 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:46.940124    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:47.028988 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.028968    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:47:47.039321 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:47.039282    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2a6a1bfd0dc2f4c5a2b0190766ed9a.slice/crio-4ce10529e6489cb63565d846b5aafb60d2f54843aa2633749a4616d102a29485 WatchSource:0}: Error finding container 4ce10529e6489cb63565d846b5aafb60d2f54843aa2633749a4616d102a29485: Status 404 returned error can't find the container with id 4ce10529e6489cb63565d846b5aafb60d2f54843aa2633749a4616d102a29485
Sep 10 14:47:47.039547 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:47.039525    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43f56a932ead032b888e9a94f11feab.slice/crio-6f2f0ae4e41a63380facebd051317691867b985b6a59c95e2f00f727c552ceb9 WatchSource:0}: Error finding container 6f2f0ae4e41a63380facebd051317691867b985b6a59c95e2f00f727c552ceb9: Status 404 returned error can't find the container with id 6f2f0ae4e41a63380facebd051317691867b985b6a59c95e2f00f727c552ceb9
Sep 10 14:47:47.040590 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.040571    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:47.044129 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.044113    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:47:47.141587 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.141526    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:47.241979 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.241941    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:47.342632 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.342596    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-2.ec2.internal\" not found"
Sep 10 14:47:47.365728 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.365707    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:47:47.442570 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.442514    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:47.457795 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.457764    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:47:47.458743 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.458725    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal"
Sep 10 14:47:47.468796 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.468772    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:47:47.815491 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.815271    2565 apiserver.go:52] "Watching apiserver"
Sep 10 14:47:47.824878 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.824846    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 14:47:47.825809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.825777    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vbtcs","openshift-dns/node-resolver-xzqgt","openshift-multus/multus-additional-cni-plugins-swf9r","openshift-multus/network-metrics-daemon-2vr4h","openshift-network-diagnostics/network-check-target-mn2b9","openshift-network-operator/iptables-alerter-bdldg","openshift-ovn-kubernetes/ovnkube-node-sxl49","kube-system/konnectivity-agent-dww8c","kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc","openshift-cluster-node-tuning-operator/tuned-zsrbk","openshift-image-registry/node-ca-8d9nt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal","openshift-multus/multus-jp4tp"]
Sep 10 14:47:47.828695 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.828670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.831180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.830618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.831671 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.831443    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 14:47:47.831671 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.831449    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 14:47:47.832935 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.832467    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vv7f\""
Sep 10 14:47:47.832935 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.832535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 14:47:47.832935 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.832594    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 14:47:47.832935 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.832540    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 14:47:47.832935 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.832849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 14:47:47.833216 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.832946    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.833385 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.833366    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:47:47.833573 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.833558    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 14:47:47.833631 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.833618    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hs8qf\""
Sep 10 14:47:47.835098 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.835076    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 14:47:47.835183 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.835155    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:47.835261 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.835236    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:47:47.835768 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.835597    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 14:47:47.835768 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.835615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-49sb6\""
Sep 10 14:47:47.835921 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.835838    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 14:47:47.835975 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.835963    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 14:47:47.836023 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.835998    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 14:47:47.837569 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.837551    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:47.837686 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.837592    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:47:47.840036 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.840011    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.842322 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.842305    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 14:47:47.842417 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.842326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.843708 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.843690    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 14:47:47.843803 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.843690    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qn8b9\""
Sep 10 14:47:47.843803 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.843727    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 14:47:47.843901 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.843874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:47:47.844610 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.844455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jz7xn\""
Sep 10 14:47:47.844716 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.844611    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 14:47:47.844716 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.844705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:47.847258 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.847236    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 14:47:47.847347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.847287    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 14:47:47.847411 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.847344    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mx8l5\""
Sep 10 14:47:47.847536 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.847520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:47.849474 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.849455    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 14:47:47.849604 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.849589    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 14:47:47.850190 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.849928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxl9b\""
Sep 10 14:47:47.850273 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.850224    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.852796 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.852386    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 14:47:47.852796 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.852617    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 14:47:47.852796 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.852722    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:47:47.852796 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.852725    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 14:47:47.852796 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.852762    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zsrsg\""
Sep 10 14:47:47.853072 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.852838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:47.853072 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.852903    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:47:47.855711 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.855007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:47.857284 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.853106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-slash\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.857378 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-run-netns\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.857378 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-var-lib-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.857492 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6q9m\" (UniqueName: \"kubernetes.io/projected/cbae3109-7439-48ce-8ad5-4f88113a4300-kube-api-access-s6q9m\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.857492 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.857592 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f2cx\" (UniqueName: \"kubernetes.io/projected/31b838c8-914a-4f54-9a9d-016faee82e93-kube-api-access-9f2cx\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.857592 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-hostroot\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.857709 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.857709 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-tuned\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.857709 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b2c2-0f8c-45c7-b712-46810b3c5f88-konnectivity-ca\") pod \"konnectivity-agent-dww8c\" (UID: \"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88\") " pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:47.857709 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857703    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-k8s-cni-cncf-io\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.857898 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysctl-conf\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.857898 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cnibin\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.857898 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857791    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-netns\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.857898 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-cni-bin\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.857898 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-sys\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.857898 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-run-ovn-kubernetes\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-kubernetes\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-systemd\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-system-cni-dir\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.857997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-socket-dir-parent\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-systemd-units\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-host\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-conf-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-node-log\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.858180 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-var-lib-kubelet\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31b838c8-914a-4f54-9a9d-016faee82e93-host-slash\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-kubelet\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-multus-certs\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fgjfj\""
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-etc-kubernetes\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysctl-d\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-run\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-cni-netd\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8wlp\" (UniqueName: \"kubernetes.io/projected/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-kube-api-access-n8wlp\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a0f538b-d498-4059-924a-212cbf31c646-multus-daemon-config\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jlt\" (UniqueName: \"kubernetes.io/projected/d5c70165-58c3-4310-b7c6-c2c78db98878-kube-api-access-76jlt\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858553    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-cni-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-cnibin\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.858635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c70165-58c3-4310-b7c6-c2c78db98878-tmp\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31b838c8-914a-4f54-9a9d-016faee82e93-iptables-alerter-script\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-system-cni-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-log-socket\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858791    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-ovnkube-config\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858813    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbae3109-7439-48ce-8ad5-4f88113a4300-ovn-node-metrics-cert\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-ovnkube-script-lib\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-modprobe-d\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysconfig\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-lib-modules\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-cni-bin\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nbv\" (UniqueName: \"kubernetes.io/projected/1a0f538b-d498-4059-924a-212cbf31c646-kube-api-access-z4nbv\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-kubelet\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.858497    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 14:47:47.859432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-systemd\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859168    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-etc-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-env-overrides\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8cd4b2c2-0f8c-45c7-b712-46810b3c5f88-agent-certs\") pod \"konnectivity-agent-dww8c\" (UID: \"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88\") " pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a0f538b-d498-4059-924a-212cbf31c646-cni-binary-copy\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-ovn\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmsds\" (UniqueName: \"kubernetes.io/projected/0016d121-63c2-4d58-8110-81f9d47e1a64-kube-api-access-vmsds\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:47.860290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.859385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-cni-multus\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.899479 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.899450    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-89glz"
Sep 10 14:47:47.906872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.906806    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-89glz"
Sep 10 14:47:47.943590 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.943565    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 14:47:47.959619 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysctl-conf\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.959619 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cnibin\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-netns\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229n6\" (UniqueName: \"kubernetes.io/projected/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-kube-api-access-229n6\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-cni-bin\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cnibin\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-sys\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-run-ovn-kubernetes\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-sys\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.959809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-kubernetes\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-systemd\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-netns\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-system-cni-dir\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-cni-bin\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-socket-dir-parent\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-run-ovn-kubernetes\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-tmp-dir\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-systemd-units\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-host\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysctl-conf\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.959998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-conf-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-kubernetes\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-etc-selinux\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-conf-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-node-log\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.960192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-var-lib-kubelet\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-socket-dir-parent\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31b838c8-914a-4f54-9a9d-016faee82e93-host-slash\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-system-cni-dir\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-kubelet\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-systemd\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-multus-certs\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-multus-certs\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-host\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-etc-kubernetes\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31b838c8-914a-4f54-9a9d-016faee82e93-host-slash\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-sys-fs\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-node-log\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-var-lib-kubelet\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysctl-d\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-kubelet\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-etc-kubernetes\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.960977 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-systemd-units\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-run\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-registration-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-cni-netd\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysctl-d\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-run\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-cni-netd\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8wlp\" (UniqueName: \"kubernetes.io/projected/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-kube-api-access-n8wlp\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a0f538b-d498-4059-924a-212cbf31c646-multus-daemon-config\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f8134df-e311-49b1-bbeb-59171cd288eb-serviceca\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4c8\" (UniqueName: \"kubernetes.io/projected/9f8134df-e311-49b1-bbeb-59171cd288eb-kube-api-access-fs4c8\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jlt\" (UniqueName: \"kubernetes.io/projected/d5c70165-58c3-4310-b7c6-c2c78db98878-kube-api-access-76jlt\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-cni-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-cnibin\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.961933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c70165-58c3-4310-b7c6-c2c78db98878-tmp\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31b838c8-914a-4f54-9a9d-016faee82e93-iptables-alerter-script\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-cnibin\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-system-cni-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-dbus\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-multus-cni-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-device-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f8134df-e311-49b1-bbeb-59171cd288eb-host\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-log-socket\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-ovnkube-config\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbae3109-7439-48ce-8ad5-4f88113a4300-ovn-node-metrics-cert\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-ovnkube-script-lib\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-modprobe-d\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-socket-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.962766 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysconfig\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.960990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-lib-modules\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-cni-bin\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nbv\" (UniqueName: \"kubernetes.io/projected/1a0f538b-d498-4059-924a-212cbf31c646-kube-api-access-z4nbv\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-hosts-file\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a0f538b-d498-4059-924a-212cbf31c646-multus-daemon-config\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfsxk\" (UniqueName: \"kubernetes.io/projected/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-kube-api-access-qfsxk\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-kubelet\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-system-cni-dir\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-systemd\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-etc-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-env-overrides\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8cd4b2c2-0f8c-45c7-b712-46810b3c5f88-agent-certs\") pod \"konnectivity-agent-dww8c\" (UID: \"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88\") " pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a0f538b-d498-4059-924a-212cbf31c646-cni-binary-copy\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-ovn\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:47.963625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmsds\" (UniqueName: \"kubernetes.io/projected/0016d121-63c2-4d58-8110-81f9d47e1a64-kube-api-access-vmsds\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-cni-multus\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-kubelet-config\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-slash\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-run-netns\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-var-lib-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6q9m\" (UniqueName: \"kubernetes.io/projected/cbae3109-7439-48ce-8ad5-4f88113a4300-kube-api-access-s6q9m\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961429    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-ovnkube-config\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f2cx\" (UniqueName: \"kubernetes.io/projected/31b838c8-914a-4f54-9a9d-016faee82e93-kube-api-access-9f2cx\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-hostroot\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-sysconfig\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.961542    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.961600    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs podName:0016d121-63c2-4d58-8110-81f9d47e1a64 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:48.461571013 +0000 UTC m=+3.031110249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs") pod "network-metrics-daemon-2vr4h" (UID: "0016d121-63c2-4d58-8110-81f9d47e1a64") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-cni-bin\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.964483 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-var-lib-cni-multus\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962008    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31b838c8-914a-4f54-9a9d-016faee82e93-iptables-alerter-script\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-kubelet\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-systemd\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-log-socket\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-slash\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-run-netns\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-etc-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-var-lib-openvswitch\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-env-overrides\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.962862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-lib-modules\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-run-ovn\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-hostroot\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-modprobe-d\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.965321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.961525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbae3109-7439-48ce-8ad5-4f88113a4300-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-tuned\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b2c2-0f8c-45c7-b712-46810b3c5f88-konnectivity-ca\") pod \"konnectivity-agent-dww8c\" (UID: \"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88\") " pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-k8s-cni-cncf-io\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cbae3109-7439-48ce-8ad5-4f88113a4300-ovnkube-script-lib\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a0f538b-d498-4059-924a-212cbf31c646-host-run-k8s-cni-cncf-io\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.963550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a0f538b-d498-4059-924a-212cbf31c646-cni-binary-copy\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.964356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b2c2-0f8c-45c7-b712-46810b3c5f88-konnectivity-ca\") pod \"konnectivity-agent-dww8c\" (UID: \"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88\") " pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.965494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c70165-58c3-4310-b7c6-c2c78db98878-tmp\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.965536    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5c70165-58c3-4310-b7c6-c2c78db98878-etc-tuned\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.966192 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.965915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8cd4b2c2-0f8c-45c7-b712-46810b3c5f88-agent-certs\") pod \"konnectivity-agent-dww8c\" (UID: \"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88\") " pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:47.967980 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.967958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbae3109-7439-48ce-8ad5-4f88113a4300-ovn-node-metrics-cert\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.968417 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.968396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8wlp\" (UniqueName: \"kubernetes.io/projected/d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d-kube-api-access-n8wlp\") pod \"multus-additional-cni-plugins-swf9r\" (UID: \"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d\") " pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:47.969773 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.969751    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:47:47.969867 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.969776    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:47:47.969867 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.969791    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6bqrq for pod openshift-network-diagnostics/network-check-target-mn2b9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:47.969974 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:47.969904    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq podName:4dc9c946-3156-4721-b715-4b0045879e99 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:48.469881885 +0000 UTC m=+3.039421136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6bqrq" (UniqueName: "kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq") pod "network-check-target-mn2b9" (UID: "4dc9c946-3156-4721-b715-4b0045879e99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:47.972528 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.972497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jlt\" (UniqueName: \"kubernetes.io/projected/d5c70165-58c3-4310-b7c6-c2c78db98878-kube-api-access-76jlt\") pod \"tuned-zsrbk\" (UID: \"d5c70165-58c3-4310-b7c6-c2c78db98878\") " pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:47.974531 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.974504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmsds\" (UniqueName: \"kubernetes.io/projected/0016d121-63c2-4d58-8110-81f9d47e1a64-kube-api-access-vmsds\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:47.977143 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.977123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nbv\" (UniqueName: \"kubernetes.io/projected/1a0f538b-d498-4059-924a-212cbf31c646-kube-api-access-z4nbv\") pod \"multus-jp4tp\" (UID: \"1a0f538b-d498-4059-924a-212cbf31c646\") " pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:47.978128 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.978102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f2cx\" (UniqueName: \"kubernetes.io/projected/31b838c8-914a-4f54-9a9d-016faee82e93-kube-api-access-9f2cx\") pod \"iptables-alerter-bdldg\" (UID: \"31b838c8-914a-4f54-9a9d-016faee82e93\") " pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:47.978210 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.978150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6q9m\" (UniqueName: \"kubernetes.io/projected/cbae3109-7439-48ce-8ad5-4f88113a4300-kube-api-access-s6q9m\") pod \"ovnkube-node-sxl49\" (UID: \"cbae3109-7439-48ce-8ad5-4f88113a4300\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:47.978657 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.978592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal" event={"ID":"bf2a6a1bfd0dc2f4c5a2b0190766ed9a","Type":"ContainerStarted","Data":"4ce10529e6489cb63565d846b5aafb60d2f54843aa2633749a4616d102a29485"}
Sep 10 14:47:47.981288 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:47.981263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal" event={"ID":"e43f56a932ead032b888e9a94f11feab","Type":"ContainerStarted","Data":"6f2f0ae4e41a63380facebd051317691867b985b6a59c95e2f00f727c552ceb9"}
Sep 10 14:47:48.063786 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-tmp-dir\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:48.063932 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-etc-selinux\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.063932 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-sys-fs\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.063932 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-registration-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.063932 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f8134df-e311-49b1-bbeb-59171cd288eb-serviceca\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:48.063932 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4c8\" (UniqueName: \"kubernetes.io/projected/9f8134df-e311-49b1-bbeb-59171cd288eb-kube-api-access-fs4c8\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:48.063932 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-dbus\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-sys-fs\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-registration-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.063952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-device-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064006    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-device-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f8134df-e311-49b1-bbeb-59171cd288eb-host\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-etc-selinux\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064060    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-tmp-dir\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9f8134df-e311-49b1-bbeb-59171cd288eb-host\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.064140    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-dbus\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.064204    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret podName:2fbf0051-5f2c-4ef2-af7b-c00c1b52241d nodeName:}" failed. No retries permitted until 2026-09-10 14:47:48.564184774 +0000 UTC m=+3.133724024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret") pod "global-pull-secret-syncer-vbtcs" (UID: "2fbf0051-5f2c-4ef2-af7b-c00c1b52241d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:48.064224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-socket-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-hosts-file\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfsxk\" (UniqueName: \"kubernetes.io/projected/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-kube-api-access-qfsxk\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-socket-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-hosts-file\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9f8134df-e311-49b1-bbeb-59171cd288eb-serviceca\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-kubelet-config\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-kubelet-config\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229n6\" (UniqueName: \"kubernetes.io/projected/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-kube-api-access-229n6\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.064776 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.064583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.072910 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.072845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-229n6\" (UniqueName: \"kubernetes.io/projected/6a8ba8b5-0b8d-4485-a9d9-8396d465cf18-kube-api-access-229n6\") pod \"aws-ebs-csi-driver-node-zp2nc\" (UID: \"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.073011 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.072957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfsxk\" (UniqueName: \"kubernetes.io/projected/ae64e0a9-6159-40ff-879c-f7ff9ce3097e-kube-api-access-qfsxk\") pod \"node-resolver-xzqgt\" (UID: \"ae64e0a9-6159-40ff-879c-f7ff9ce3097e\") " pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:48.073182 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.073163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4c8\" (UniqueName: \"kubernetes.io/projected/9f8134df-e311-49b1-bbeb-59171cd288eb-kube-api-access-fs4c8\") pod \"node-ca-8d9nt\" (UID: \"9f8134df-e311-49b1-bbeb-59171cd288eb\") " pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:48.142236 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.142203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:47:48.150433 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.150416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsrbk"
Sep 10 14:47:48.162873 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.162841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swf9r"
Sep 10 14:47:48.168372 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.168351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdldg"
Sep 10 14:47:48.174966 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.174949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jp4tp"
Sep 10 14:47:48.182482 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.182459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:47:48.188990 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.188973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xzqgt"
Sep 10 14:47:48.196505 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.196487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc"
Sep 10 14:47:48.200958 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.200936    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8d9nt"
Sep 10 14:47:48.278724 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.278694    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:47:48.466782 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.466698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:48.466928 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.466860    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:48.466928 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.466924    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs podName:0016d121-63c2-4d58-8110-81f9d47e1a64 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:49.46690453 +0000 UTC m=+4.036443766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs") pod "network-metrics-daemon-2vr4h" (UID: "0016d121-63c2-4d58-8110-81f9d47e1a64") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:48.567960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.567922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:48.568124 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.568075    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:48.568124 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.568103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:48.568209 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.568140    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret podName:2fbf0051-5f2c-4ef2-af7b-c00c1b52241d nodeName:}" failed. No retries permitted until 2026-09-10 14:47:49.568121081 +0000 UTC m=+4.137660320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret") pod "global-pull-secret-syncer-vbtcs" (UID: "2fbf0051-5f2c-4ef2-af7b-c00c1b52241d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:48.568209 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.568199    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:47:48.568311 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.568219    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:47:48.568311 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.568232    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6bqrq for pod openshift-network-diagnostics/network-check-target-mn2b9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:48.568311 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.568292    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq podName:4dc9c946-3156-4721-b715-4b0045879e99 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:49.568274932 +0000 UTC m=+4.137814171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bqrq" (UniqueName: "kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq") pod "network-check-target-mn2b9" (UID: "4dc9c946-3156-4721-b715-4b0045879e99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:48.895625 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:48.895460    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b838c8_914a_4f54_9a9d_016faee82e93.slice/crio-f2a533bf79b67b87a79e0b2da3ba69318b108ae279c2fd461508e6cecf77da94 WatchSource:0}: Error finding container f2a533bf79b67b87a79e0b2da3ba69318b108ae279c2fd461508e6cecf77da94: Status 404 returned error can't find the container with id f2a533bf79b67b87a79e0b2da3ba69318b108ae279c2fd461508e6cecf77da94
Sep 10 14:47:48.897667 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:48.897629    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8134df_e311_49b1_bbeb_59171cd288eb.slice/crio-734abded2a577247566e43925fa579498aa481d9c249485d2a6e00757f3eef88 WatchSource:0}: Error finding container 734abded2a577247566e43925fa579498aa481d9c249485d2a6e00757f3eef88: Status 404 returned error can't find the container with id 734abded2a577247566e43925fa579498aa481d9c249485d2a6e00757f3eef88
Sep 10 14:47:48.899067 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:48.898535    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0f538b_d498_4059_924a_212cbf31c646.slice/crio-813c154bd5ee98eaed89556b8a98fa0daaf22270c83c8c1d32d81f2c65090ac2 WatchSource:0}: Error finding container 813c154bd5ee98eaed89556b8a98fa0daaf22270c83c8c1d32d81f2c65090ac2: Status 404 returned error can't find the container with id 813c154bd5ee98eaed89556b8a98fa0daaf22270c83c8c1d32d81f2c65090ac2
Sep 10 14:47:48.901829 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:48.901723    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8ba8b5_0b8d_4485_a9d9_8396d465cf18.slice/crio-3ac1fc400618fbdbd60e62a2b12ca8658513e14cd0512e08c3f3dd146466f168 WatchSource:0}: Error finding container 3ac1fc400618fbdbd60e62a2b12ca8658513e14cd0512e08c3f3dd146466f168: Status 404 returned error can't find the container with id 3ac1fc400618fbdbd60e62a2b12ca8658513e14cd0512e08c3f3dd146466f168
Sep 10 14:47:48.902481 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:48.902459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae64e0a9_6159_40ff_879c_f7ff9ce3097e.slice/crio-c07160b2190c993734137d628278226b713092813d6e3c0e8901da43b2d4e583 WatchSource:0}: Error finding container c07160b2190c993734137d628278226b713092813d6e3c0e8901da43b2d4e583: Status 404 returned error can't find the container with id c07160b2190c993734137d628278226b713092813d6e3c0e8901da43b2d4e583
Sep 10 14:47:48.903782 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:47:48.903398    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f7763c_17c3_4eb7_9cf2_33e5e9ffca5d.slice/crio-3cbe4d772633d2228c274883692858df624e3daceac53bab7a96175b48dca9fa WatchSource:0}: Error finding container 3cbe4d772633d2228c274883692858df624e3daceac53bab7a96175b48dca9fa: Status 404 returned error can't find the container with id 3cbe4d772633d2228c274883692858df624e3daceac53bab7a96175b48dca9fa
Sep 10 14:47:48.911450 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.911423    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:42:47 +0000 UTC" deadline="2028-05-06 20:12:22.09893187 +0000 UTC"
Sep 10 14:47:48.911450 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.911449    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14501h24m33.187486547s"
Sep 10 14:47:48.970611 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.970590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:48.970730 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:48.970710    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:47:48.985309 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.985277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc" event={"ID":"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18","Type":"ContainerStarted","Data":"3ac1fc400618fbdbd60e62a2b12ca8658513e14cd0512e08c3f3dd146466f168"}
Sep 10 14:47:48.986431 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.986408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jp4tp" event={"ID":"1a0f538b-d498-4059-924a-212cbf31c646","Type":"ContainerStarted","Data":"813c154bd5ee98eaed89556b8a98fa0daaf22270c83c8c1d32d81f2c65090ac2"}
Sep 10 14:47:48.989934 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.989912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dww8c" event={"ID":"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88","Type":"ContainerStarted","Data":"7b511d932aaa31019a8e845eba69c981281a165cb9050f20ae23489c79256af5"}
Sep 10 14:47:48.994351 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.994330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8d9nt" event={"ID":"9f8134df-e311-49b1-bbeb-59171cd288eb","Type":"ContainerStarted","Data":"734abded2a577247566e43925fa579498aa481d9c249485d2a6e00757f3eef88"}
Sep 10 14:47:48.998523 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:48.998501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdldg" event={"ID":"31b838c8-914a-4f54-9a9d-016faee82e93","Type":"ContainerStarted","Data":"f2a533bf79b67b87a79e0b2da3ba69318b108ae279c2fd461508e6cecf77da94"}
Sep 10 14:47:49.003848 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.003823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsrbk" event={"ID":"d5c70165-58c3-4310-b7c6-c2c78db98878","Type":"ContainerStarted","Data":"4a6d73c5b2fd8da512749122a6e6f332a5d15cddf2da5234490eb4f7316f02a0"}
Sep 10 14:47:49.005226 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.005208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"bef695cbb1a8b9b1d85c39d012c48dcdf3befaa376825315e7b70445597d0d65"}
Sep 10 14:47:49.008033 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.008004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerStarted","Data":"3cbe4d772633d2228c274883692858df624e3daceac53bab7a96175b48dca9fa"}
Sep 10 14:47:49.011558 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.011539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xzqgt" event={"ID":"ae64e0a9-6159-40ff-879c-f7ff9ce3097e","Type":"ContainerStarted","Data":"c07160b2190c993734137d628278226b713092813d6e3c0e8901da43b2d4e583"}
Sep 10 14:47:49.016846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.016807    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal" podStartSLOduration=2.0167959030000002 podStartE2EDuration="2.016795903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:49.016416197 +0000 UTC m=+3.585955455" watchObservedRunningTime="2026-09-10 14:47:49.016795903 +0000 UTC m=+3.586335161"
Sep 10 14:47:49.473140 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.473053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:49.473289 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.473267    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:49.473344 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.473331    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs podName:0016d121-63c2-4d58-8110-81f9d47e1a64 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:51.473312323 +0000 UTC m=+6.042851574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs") pod "network-metrics-daemon-2vr4h" (UID: "0016d121-63c2-4d58-8110-81f9d47e1a64") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:49.573929 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.573883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:49.574105 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.573954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:49.574105 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.574100    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:47:49.574219 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.574122    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:47:49.574219 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.574134    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6bqrq for pod openshift-network-diagnostics/network-check-target-mn2b9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:49.574219 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.574194    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq podName:4dc9c946-3156-4721-b715-4b0045879e99 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:51.574175536 +0000 UTC m=+6.143714774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bqrq" (UniqueName: "kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq") pod "network-check-target-mn2b9" (UID: "4dc9c946-3156-4721-b715-4b0045879e99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:49.574619 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.574600    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:49.574729 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.574668    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret podName:2fbf0051-5f2c-4ef2-af7b-c00c1b52241d nodeName:}" failed. No retries permitted until 2026-09-10 14:47:51.574652984 +0000 UTC m=+6.144192260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret") pod "global-pull-secret-syncer-vbtcs" (UID: "2fbf0051-5f2c-4ef2-af7b-c00c1b52241d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:49.911806 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.911673    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:42:47 +0000 UTC" deadline="2028-06-10 05:47:08.47298733 +0000 UTC"
Sep 10 14:47:49.911806 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.911729    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15326h59m18.561262559s"
Sep 10 14:47:49.981176 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.981150    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:49.981176 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:49.981170    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:49.981384 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.981290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:47:49.981384 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:49.981366    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:47:50.025805 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:50.024784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-2.ec2.internal" event={"ID":"bf2a6a1bfd0dc2f4c5a2b0190766ed9a","Type":"ContainerStarted","Data":"9f6106aaa13663050da8093cfdba8edc661cc173845bfcf95510b86c9af7e6df"}
Sep 10 14:47:50.029664 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:50.029617    2565 generic.go:356] "Generic (PLEG): container finished" podID="e43f56a932ead032b888e9a94f11feab" containerID="458e433a62fe23471d0da2df0a074f1c40d9f6ece5527db06c594d86cfb7c99d" exitCode=0
Sep 10 14:47:50.029786 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:50.029669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal" event={"ID":"e43f56a932ead032b888e9a94f11feab","Type":"ContainerDied","Data":"458e433a62fe23471d0da2df0a074f1c40d9f6ece5527db06c594d86cfb7c99d"}
Sep 10 14:47:50.972105 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:50.971519    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:50.972105 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:50.971725    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:47:51.047527 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:51.047428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal" event={"ID":"e43f56a932ead032b888e9a94f11feab","Type":"ContainerStarted","Data":"7b3269a83c462fbac50bd98a3000d40c89bd743066821c096454fd3149c4f49a"}
Sep 10 14:47:51.505083 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:51.504991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:51.505254 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.505169    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:51.505254 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.505231    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs podName:0016d121-63c2-4d58-8110-81f9d47e1a64 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:55.5052131 +0000 UTC m=+10.074752341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs") pod "network-metrics-daemon-2vr4h" (UID: "0016d121-63c2-4d58-8110-81f9d47e1a64") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:51.606265 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:51.606225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:51.606455 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:51.606297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:51.606522 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.606470    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:47:51.606522 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.606488    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:47:51.606522 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.606501    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6bqrq for pod openshift-network-diagnostics/network-check-target-mn2b9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:51.606683 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.606560    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq podName:4dc9c946-3156-4721-b715-4b0045879e99 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:55.606541335 +0000 UTC m=+10.176080587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bqrq" (UniqueName: "kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq") pod "network-check-target-mn2b9" (UID: "4dc9c946-3156-4721-b715-4b0045879e99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:51.607045 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.606957    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:51.607045 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.607007    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret podName:2fbf0051-5f2c-4ef2-af7b-c00c1b52241d nodeName:}" failed. No retries permitted until 2026-09-10 14:47:55.606992903 +0000 UTC m=+10.176532146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret") pod "global-pull-secret-syncer-vbtcs" (UID: "2fbf0051-5f2c-4ef2-af7b-c00c1b52241d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:51.971620 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:51.971041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:51.971620 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.971185    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:47:51.971620 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:51.971585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:51.971903 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:51.971689    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:47:52.971758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:52.971230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:52.971758 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:52.971362    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:47:53.970510 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:53.970480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:53.970713 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:53.970480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:53.970713 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:53.970613    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:47:53.970839 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:53.970711    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:47:54.970731 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:54.970635    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:54.971149 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:54.970785    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:47:55.542234 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:55.542194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:55.542442 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.542351    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:55.542740 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.542720    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs podName:0016d121-63c2-4d58-8110-81f9d47e1a64 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:03.542693012 +0000 UTC m=+18.112232302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs") pod "network-metrics-daemon-2vr4h" (UID: "0016d121-63c2-4d58-8110-81f9d47e1a64") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:47:55.642816 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:55.642777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:55.643008 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:55.642860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:55.643008 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.642960    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:55.643126 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.643010    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:47:55.643126 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.643027    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:47:55.643126 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.643033    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret podName:2fbf0051-5f2c-4ef2-af7b-c00c1b52241d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:03.643013297 +0000 UTC m=+18.212552537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret") pod "global-pull-secret-syncer-vbtcs" (UID: "2fbf0051-5f2c-4ef2-af7b-c00c1b52241d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:47:55.643126 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.643042    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6bqrq for pod openshift-network-diagnostics/network-check-target-mn2b9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:55.643126 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.643091    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq podName:4dc9c946-3156-4721-b715-4b0045879e99 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:03.643075742 +0000 UTC m=+18.212614980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bqrq" (UniqueName: "kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq") pod "network-check-target-mn2b9" (UID: "4dc9c946-3156-4721-b715-4b0045879e99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:47:55.971542 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:55.971459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:55.972002 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.971587    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:47:55.972002 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:55.971658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:55.972002 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:55.971768    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:47:56.971725 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:56.971200    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:56.971725 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:56.971358    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:47:57.971517 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:57.971484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:57.971728 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:57.971532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:57.971728 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:57.971661    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:47:57.972133 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:57.971760    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:47:58.970547 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:58.970512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:47:58.970770 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:58.970654    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:47:59.970835 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:59.970545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:47:59.970835 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:47:59.970545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:47:59.971307 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:59.970895    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:47:59.971307 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:47:59.971017    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:00.971150 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:00.971118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:00.971544 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:00.971223    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:01.970761 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:01.970708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:01.970761 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:01.970755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:01.970996 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:01.970836    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:01.971051 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:01.970995    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:02.971407 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:02.971372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:02.971916 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:02.971498    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:03.603086 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:03.603000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:03.603251 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.603164    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:48:03.603314 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.603263    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs podName:0016d121-63c2-4d58-8110-81f9d47e1a64 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:19.60323943 +0000 UTC m=+34.172778679 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs") pod "network-metrics-daemon-2vr4h" (UID: "0016d121-63c2-4d58-8110-81f9d47e1a64") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:48:03.703851 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:03.703816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:03.704023 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:03.703873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:03.704023 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.703975    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:48:03.704141 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.704054    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret podName:2fbf0051-5f2c-4ef2-af7b-c00c1b52241d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:19.704033572 +0000 UTC m=+34.273572809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret") pod "global-pull-secret-syncer-vbtcs" (UID: "2fbf0051-5f2c-4ef2-af7b-c00c1b52241d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:48:03.704141 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.703987    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:48:03.704141 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.704089    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:48:03.704141 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.704114    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6bqrq for pod openshift-network-diagnostics/network-check-target-mn2b9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:48:03.704323 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.704161    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq podName:4dc9c946-3156-4721-b715-4b0045879e99 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:19.704148261 +0000 UTC m=+34.273687511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bqrq" (UniqueName: "kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq") pod "network-check-target-mn2b9" (UID: "4dc9c946-3156-4721-b715-4b0045879e99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:48:03.971264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:03.971179    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:03.971425 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:03.971183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:03.971425 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.971321    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:03.971425 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:03.971364    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:04.971214 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:04.971176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:04.971411 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:04.971306    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:05.971788 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:05.971761    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:05.972244 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:05.971866    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:05.972244 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:05.972030    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:05.972244 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:05.972100    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:06.970633 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:06.970597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:06.970802 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:06.970775    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:07.084741 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.084704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"5e862b3bd0790646c45a1e1f85aeee132d8eb4ab4b7329e673775ec175015d20"}
Sep 10 14:48:07.085434 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.084780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"b2a94f71398cb14e6a7e94920120bb558aeb282448e2a331aa27e0e51dd6907b"}
Sep 10 14:48:07.085434 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.084825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"d1a6a39941f36df9a64b00e28a6fcfd72484229c38d91f6467ae7718d85aa41b"}
Sep 10 14:48:07.086256 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.086233    2565 generic.go:356] "Generic (PLEG): container finished" podID="d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d" containerID="4c956184013a3fd5c43b047a9b56e683aabe9886820bdb73e0a5780ef178056b" exitCode=0
Sep 10 14:48:07.086373 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.086297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerDied","Data":"4c956184013a3fd5c43b047a9b56e683aabe9886820bdb73e0a5780ef178056b"}
Sep 10 14:48:07.087775 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.087696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xzqgt" event={"ID":"ae64e0a9-6159-40ff-879c-f7ff9ce3097e","Type":"ContainerStarted","Data":"99a7c42b077d189359984253e61df94f09ff700b0878ac700d8b38ad7958cc12"}
Sep 10 14:48:07.089090 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.089070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc" event={"ID":"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18","Type":"ContainerStarted","Data":"750e1dc9bf7a253f931c3f7d6a30843211912fa2cde183f3d432c95bc7b1efa4"}
Sep 10 14:48:07.090628 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.090606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jp4tp" event={"ID":"1a0f538b-d498-4059-924a-212cbf31c646","Type":"ContainerStarted","Data":"f765301003e21d1af86522a25b37076d4a0cc2c5cfedef3590f75d996bb705ae"}
Sep 10 14:48:07.092063 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.092044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dww8c" event={"ID":"8cd4b2c2-0f8c-45c7-b712-46810b3c5f88","Type":"ContainerStarted","Data":"a557eb9a3e624ad121163edb4d4756bb8999538c3da7c54ec8ae720457937c00"}
Sep 10 14:48:07.093508 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.093485    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8d9nt" event={"ID":"9f8134df-e311-49b1-bbeb-59171cd288eb","Type":"ContainerStarted","Data":"349d430734759232bbd9a5e18595710ba058ec68117aebb809eea314e14ece0b"}
Sep 10 14:48:07.094824 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.094803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsrbk" event={"ID":"d5c70165-58c3-4310-b7c6-c2c78db98878","Type":"ContainerStarted","Data":"30714732bf1bf06765b7df3410a2d8c28f53ecb76a40028d65b179ad1f04e48c"}
Sep 10 14:48:07.106122 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.106071    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-2.ec2.internal" podStartSLOduration=20.106051264 podStartE2EDuration="20.106051264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:51.06309492 +0000 UTC m=+5.632634179" watchObservedRunningTime="2026-09-10 14:48:07.106051264 +0000 UTC m=+21.675590522"
Sep 10 14:48:07.131920 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.131875    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dww8c" podStartSLOduration=6.770739418 podStartE2EDuration="21.131854732s" totalImagesPullingTime="14.361115314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:07.119611731 +0000 UTC m=+21.689150989" watchObservedRunningTime="2026-09-10 14:48:07.131854732 +0000 UTC m=+21.701393991"
Sep 10 14:48:07.132160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.132133    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8d9nt" podStartSLOduration=3.410476848 podStartE2EDuration="21.132124401s" totalImagesPullingTime="17.721647553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:07.131998248 +0000 UTC m=+21.701537497" watchObservedRunningTime="2026-09-10 14:48:07.132124401 +0000 UTC m=+21.701663659"
Sep 10 14:48:07.146926 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.146881    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zsrbk" podStartSLOduration=3.420025918 podStartE2EDuration="21.146866803s" totalImagesPullingTime="17.726840885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:07.146435891 +0000 UTC m=+21.715975148" watchObservedRunningTime="2026-09-10 14:48:07.146866803 +0000 UTC m=+21.716406060"
Sep 10 14:48:07.165334 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.165281    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jp4tp" podStartSLOduration=3.378127686 podStartE2EDuration="21.165260727s" totalImagesPullingTime="17.787133041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:07.165044288 +0000 UTC m=+21.734583547" watchObservedRunningTime="2026-09-10 14:48:07.165260727 +0000 UTC m=+21.734799987"
Sep 10 14:48:07.178577 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.178536    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xzqgt" podStartSLOduration=6.812844077 podStartE2EDuration="21.178522053s" totalImagesPullingTime="14.365677976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:07.178186353 +0000 UTC m=+21.747725610" watchObservedRunningTime="2026-09-10 14:48:07.178522053 +0000 UTC m=+21.748061310"
Sep 10 14:48:07.970820 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.970591    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:07.970998 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:07.970670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:07.970998 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:07.970910    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:07.971134 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:07.971008    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:08.101716 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.101670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"5f69ce0049736d09c8e48fb92edaef0bea6039ef337934414e95a7b978394d45"}
Sep 10 14:48:08.102093 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.101762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"9f6d0a22e40b8a5df053ac27d1bbe7d683c90c2b1745ffa12ce603ca7d636b9a"}
Sep 10 14:48:08.102093 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.101813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"08a390750589a5e8597bdd256945b9a569b94466ef3dff3157a0d7f9021c6027"}
Sep 10 14:48:08.103595 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.103559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdldg" event={"ID":"31b838c8-914a-4f54-9a9d-016faee82e93","Type":"ContainerStarted","Data":"319e2a83f70d2634adca4e562a4aff0358ca19ce4d253f1325f5ede97edb79eb"}
Sep 10 14:48:08.118698 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.118627    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bdldg" podStartSLOduration=4.394336927 podStartE2EDuration="22.118606946s" totalImagesPullingTime="17.724270019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:08.118388244 +0000 UTC m=+22.687927502" watchObservedRunningTime="2026-09-10 14:48:08.118606946 +0000 UTC m=+22.688146205"
Sep 10 14:48:08.226376 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.226345    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 14:48:08.519633 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.519552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:48:08.923505 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.923402    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:48:08.226370099Z","UUID":"75dcbdd6-e38f-4dd8-93e5-3caba1c1c0f9","Handler":null,"Name":"","Endpoint":""}
Sep 10 14:48:08.924931 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.924910    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 14:48:08.925039 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.924945    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 14:48:08.971292 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:08.971257    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:08.971488 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:08.971369    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:09.110347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:09.110309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc" event={"ID":"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18","Type":"ContainerStarted","Data":"3ab76da0afffecad31042693755cb8cf163998ab637e0d9d90018017dca13fd5"}
Sep 10 14:48:09.971307 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:09.971270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:09.971480 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:09.971270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:09.971480 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:09.971410    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:09.971614 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:09.971512    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:10.117079 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:10.117040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"8b6fd4875d1f8574aea02bbf86daafef069fafa83aff6864e4ffc3196dcc0326"}
Sep 10 14:48:10.465534 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:10.465499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:48:10.466155 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:10.466127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:48:10.971185 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:10.971153    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:10.971348 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:10.971271    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:11.119549 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:11.119524    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dww8c"
Sep 10 14:48:11.971020 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:11.970741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:11.971150 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:11.970760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:11.971150 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:11.971082    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:11.971150 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:11.971126    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:12.124418 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.124380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" event={"ID":"cbae3109-7439-48ce-8ad5-4f88113a4300","Type":"ContainerStarted","Data":"d505c13d55b39fe725bf04e3018525963b3e599f2c34e15bd930db0d6c170525"}
Sep 10 14:48:12.125106 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.124654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:48:12.126391 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.126361    2565 generic.go:356] "Generic (PLEG): container finished" podID="d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d" containerID="ee4aefad2ea5c7ca271847956e1209c720ba744350f2a2e1cc73f42330cddb76" exitCode=0
Sep 10 14:48:12.126484 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.126442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerDied","Data":"ee4aefad2ea5c7ca271847956e1209c720ba744350f2a2e1cc73f42330cddb76"}
Sep 10 14:48:12.128729 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.128707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc" event={"ID":"6a8ba8b5-0b8d-4485-a9d9-8396d465cf18","Type":"ContainerStarted","Data":"17d4ffab6684b4770a9dbede1b042d9dd39c012128d0996ca21ec384cf74fc75"}
Sep 10 14:48:12.139982 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.139963    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:48:12.148778 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.148743    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49" podStartSLOduration=8.369134324000001 podStartE2EDuration="26.148729654s" totalImagesPullingTime="17.77959533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:12.147217974 +0000 UTC m=+26.716757228" watchObservedRunningTime="2026-09-10 14:48:12.148729654 +0000 UTC m=+26.718268912"
Sep 10 14:48:12.187265 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.187226    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp2nc" podStartSLOduration=4.060902693 podStartE2EDuration="26.187213862s" totalImagesPullingTime="22.126311169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:12.186564268 +0000 UTC m=+26.756103526" watchObservedRunningTime="2026-09-10 14:48:12.187213862 +0000 UTC m=+26.756753119"
Sep 10 14:48:12.971470 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:12.971435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:12.971619 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:12.971558    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:13.134009 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.133831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerStarted","Data":"9fede4e9971c51939a8a95c6eaa8a0c4087115344250ec569e69cb047c115b1a"}
Sep 10 14:48:13.136702 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.135510    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:48:13.136702 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.135608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:48:13.151516 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.151466    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:48:13.498783 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.498751    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbtcs"]
Sep 10 14:48:13.498981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.498902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:13.499055 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:13.498995    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:13.501698 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.501672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mn2b9"]
Sep 10 14:48:13.501802 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.501769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:13.501880 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:13.501859    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:13.502479 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.502454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vr4h"]
Sep 10 14:48:13.502584 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:13.502558    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:13.502676 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:13.502658    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:14.138176 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:14.138145    2565 generic.go:356] "Generic (PLEG): container finished" podID="d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d" containerID="9fede4e9971c51939a8a95c6eaa8a0c4087115344250ec569e69cb047c115b1a" exitCode=0
Sep 10 14:48:14.138619 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:14.138216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerDied","Data":"9fede4e9971c51939a8a95c6eaa8a0c4087115344250ec569e69cb047c115b1a"}
Sep 10 14:48:14.970880 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:14.970845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:14.971045 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:14.970845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:14.971045 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:14.970954    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:14.971045 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:14.970845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:14.971045 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:14.971028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:14.971195 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:14.971106    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:15.145229 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:15.145198    2565 generic.go:356] "Generic (PLEG): container finished" podID="d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d" containerID="31dbc5d4db83028675d4bf0f505ec40098c14a0ce5544e362e9fbc3f49dcaf23" exitCode=0
Sep 10 14:48:15.145628 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:15.145283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerDied","Data":"31dbc5d4db83028675d4bf0f505ec40098c14a0ce5544e362e9fbc3f49dcaf23"}
Sep 10 14:48:16.970843 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:16.970806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:16.970843 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:16.970833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:16.970843 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:16.970842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:16.971304 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:16.970944    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:16.971304 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:16.971075    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:16.971304 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:16.971159    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:18.971616 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:18.971420    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:18.972128 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:18.971419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:18.972128 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:18.971724    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbtcs" podUID="2fbf0051-5f2c-4ef2-af7b-c00c1b52241d"
Sep 10 14:48:18.972128 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:18.971430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:18.972128 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:18.971765    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mn2b9" podUID="4dc9c946-3156-4721-b715-4b0045879e99"
Sep 10 14:48:18.972128 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:18.971887    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vr4h" podUID="0016d121-63c2-4d58-8110-81f9d47e1a64"
Sep 10 14:48:19.621841 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:19.621811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:19.621951 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.621924    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:48:19.621995 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.621971    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs podName:0016d121-63c2-4d58-8110-81f9d47e1a64 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:51.621958378 +0000 UTC m=+66.191497614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs") pod "network-metrics-daemon-2vr4h" (UID: "0016d121-63c2-4d58-8110-81f9d47e1a64") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:48:19.722791 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:19.722744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:19.722940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:19.722828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:19.722940 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.722885    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:48:19.723069 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.722955    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret podName:2fbf0051-5f2c-4ef2-af7b-c00c1b52241d nodeName:}" failed. No retries permitted until 2026-09-10 14:48:51.722939175 +0000 UTC m=+66.292478415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret") pod "global-pull-secret-syncer-vbtcs" (UID: "2fbf0051-5f2c-4ef2-af7b-c00c1b52241d") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:48:19.723069 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.722997    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:48:19.723069 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.723020    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:48:19.723069 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.723033    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6bqrq for pod openshift-network-diagnostics/network-check-target-mn2b9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:48:19.723214 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:19.723091    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq podName:4dc9c946-3156-4721-b715-4b0045879e99 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:51.723073135 +0000 UTC m=+66.292612389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6bqrq" (UniqueName: "kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq") pod "network-check-target-mn2b9" (UID: "4dc9c946-3156-4721-b715-4b0045879e99") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:48:20.159904 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.159870    2565 generic.go:356] "Generic (PLEG): container finished" podID="d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d" containerID="282fe61a345a52b32442b9253d13bdf4f36df8adc63912b043bdabb7a2cec156" exitCode=0
Sep 10 14:48:20.160309 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.159912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerDied","Data":"282fe61a345a52b32442b9253d13bdf4f36df8adc63912b043bdabb7a2cec156"}
Sep 10 14:48:20.718263 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.718235    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-2.ec2.internal" event="NodeReady"
Sep 10 14:48:20.718418 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.718358    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 14:48:20.769607 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.768693    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v46nj"]
Sep 10 14:48:20.789518 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.789484    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ttnzd"]
Sep 10 14:48:20.789713 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.789572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:20.791708 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.791686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 14:48:20.791828 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.791734    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 14:48:20.807468 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.807440    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7c4td\""
Sep 10 14:48:20.811527 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.811509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v46nj"]
Sep 10 14:48:20.811590 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.811542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ttnzd"]
Sep 10 14:48:20.811630 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.811603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:20.813845 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.813827    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 14:48:20.813845 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.813836    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 14:48:20.813999 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.813829    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q5d77\""
Sep 10 14:48:20.813999 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.813901    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 14:48:20.931015 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.930979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c065e49-1594-4143-8c64-d2894da542a5-config-volume\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:20.931015 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.931014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:20.931237 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.931037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a01ff9d-c190-49e0-8196-405a2df80685-cert\") pod \"ingress-canary-ttnzd\" (UID: \"9a01ff9d-c190-49e0-8196-405a2df80685\") " pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:20.931237 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.931088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c065e49-1594-4143-8c64-d2894da542a5-tmp-dir\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:20.931237 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.931113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbf6\" (UniqueName: \"kubernetes.io/projected/9a01ff9d-c190-49e0-8196-405a2df80685-kube-api-access-fvbf6\") pod \"ingress-canary-ttnzd\" (UID: \"9a01ff9d-c190-49e0-8196-405a2df80685\") " pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:20.931237 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.931144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8mp\" (UniqueName: \"kubernetes.io/projected/0c065e49-1594-4143-8c64-d2894da542a5-kube-api-access-kn8mp\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:20.971343 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.971259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:20.971343 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.971285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:20.971343 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.971259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:20.974168 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.974142    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:48:20.974298 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.974215    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2gb\""
Sep 10 14:48:20.974298 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.974251    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:48:20.974298 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.974263    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fr4wz\""
Sep 10 14:48:20.974298 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.974286    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:48:20.974491 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:20.974222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:48:21.031895 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.031870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8mp\" (UniqueName: \"kubernetes.io/projected/0c065e49-1594-4143-8c64-d2894da542a5-kube-api-access-kn8mp\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.032035 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.031931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c065e49-1594-4143-8c64-d2894da542a5-config-volume\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.032035 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.031947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.032035 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.031971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a01ff9d-c190-49e0-8196-405a2df80685-cert\") pod \"ingress-canary-ttnzd\" (UID: \"9a01ff9d-c190-49e0-8196-405a2df80685\") " pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:21.032035 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.031988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c065e49-1594-4143-8c64-d2894da542a5-tmp-dir\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.032035 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.032005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbf6\" (UniqueName: \"kubernetes.io/projected/9a01ff9d-c190-49e0-8196-405a2df80685-kube-api-access-fvbf6\") pod \"ingress-canary-ttnzd\" (UID: \"9a01ff9d-c190-49e0-8196-405a2df80685\") " pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:21.032273 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:21.032081    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:48:21.032273 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:21.032143    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls podName:0c065e49-1594-4143-8c64-d2894da542a5 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:21.532127274 +0000 UTC m=+36.101666528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls") pod "dns-default-v46nj" (UID: "0c065e49-1594-4143-8c64-d2894da542a5") : secret "dns-default-metrics-tls" not found
Sep 10 14:48:21.032422 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.032404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c065e49-1594-4143-8c64-d2894da542a5-tmp-dir\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.032592 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.032567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c065e49-1594-4143-8c64-d2894da542a5-config-volume\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.035962 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.035946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a01ff9d-c190-49e0-8196-405a2df80685-cert\") pod \"ingress-canary-ttnzd\" (UID: \"9a01ff9d-c190-49e0-8196-405a2df80685\") " pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:21.044935 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.044911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbf6\" (UniqueName: \"kubernetes.io/projected/9a01ff9d-c190-49e0-8196-405a2df80685-kube-api-access-fvbf6\") pod \"ingress-canary-ttnzd\" (UID: \"9a01ff9d-c190-49e0-8196-405a2df80685\") " pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:21.045082 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.045062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8mp\" (UniqueName: \"kubernetes.io/projected/0c065e49-1594-4143-8c64-d2894da542a5-kube-api-access-kn8mp\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.119238 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.119204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ttnzd"
Sep 10 14:48:21.167094 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.167063    2565 generic.go:356] "Generic (PLEG): container finished" podID="d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d" containerID="cbd2bd858ca097440f26b19101890b41653e1c43517ba6029bc00d9aedadf70a" exitCode=0
Sep 10 14:48:21.167697 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.167104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerDied","Data":"cbd2bd858ca097440f26b19101890b41653e1c43517ba6029bc00d9aedadf70a"}
Sep 10 14:48:21.453605 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.453441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ttnzd"]
Sep 10 14:48:21.457207 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:21.457183    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a01ff9d_c190_49e0_8196_405a2df80685.slice/crio-31f222c8ee556250ccd16d7e52105fdaf0804b3506650eab7fb5403fb3168ee6 WatchSource:0}: Error finding container 31f222c8ee556250ccd16d7e52105fdaf0804b3506650eab7fb5403fb3168ee6: Status 404 returned error can't find the container with id 31f222c8ee556250ccd16d7e52105fdaf0804b3506650eab7fb5403fb3168ee6
Sep 10 14:48:21.535341 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:21.535276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:21.535424 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:21.535381    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:48:21.535474 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:21.535428    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls podName:0c065e49-1594-4143-8c64-d2894da542a5 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:22.535412387 +0000 UTC m=+37.104951639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls") pod "dns-default-v46nj" (UID: "0c065e49-1594-4143-8c64-d2894da542a5") : secret "dns-default-metrics-tls" not found
Sep 10 14:48:22.170208 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:22.170176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ttnzd" event={"ID":"9a01ff9d-c190-49e0-8196-405a2df80685","Type":"ContainerStarted","Data":"31f222c8ee556250ccd16d7e52105fdaf0804b3506650eab7fb5403fb3168ee6"}
Sep 10 14:48:22.173309 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:22.173287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swf9r" event={"ID":"d6f7763c-17c3-4eb7-9cf2-33e5e9ffca5d","Type":"ContainerStarted","Data":"876529e7b415feebc13c28ea9332a7aa1d9e62fe79cdb925dfdeec24cba5cf18"}
Sep 10 14:48:22.195377 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:22.195335    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-swf9r" podStartSLOduration=8.218190618 podStartE2EDuration="36.195319682s" totalImagesPullingTime="27.977129064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:22.194234489 +0000 UTC m=+36.763773747" watchObservedRunningTime="2026-09-10 14:48:22.195319682 +0000 UTC m=+36.764858940"
Sep 10 14:48:22.543290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:22.543201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:22.543438 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:22.543405    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:48:22.543489 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:22.543484    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls podName:0c065e49-1594-4143-8c64-d2894da542a5 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:24.543464375 +0000 UTC m=+39.113003611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls") pod "dns-default-v46nj" (UID: "0c065e49-1594-4143-8c64-d2894da542a5") : secret "dns-default-metrics-tls" not found
Sep 10 14:48:24.560458 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:24.560428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:24.560771 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:24.560551    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:48:24.560771 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:24.560602    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls podName:0c065e49-1594-4143-8c64-d2894da542a5 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:28.560586527 +0000 UTC m=+43.130125779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls") pod "dns-default-v46nj" (UID: "0c065e49-1594-4143-8c64-d2894da542a5") : secret "dns-default-metrics-tls" not found
Sep 10 14:48:25.182085 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:25.182045    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ttnzd" event={"ID":"9a01ff9d-c190-49e0-8196-405a2df80685","Type":"ContainerStarted","Data":"8540c102ca1fd86ffd4fac049b8da9add98183dddcfadbe38b232449d50621fc"}
Sep 10 14:48:25.197615 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:25.197569    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ttnzd" podStartSLOduration=2.230556584 podStartE2EDuration="5.197555152s" totalImagesPullingTime="2.966998568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:25.196628219 +0000 UTC m=+39.766167478" watchObservedRunningTime="2026-09-10 14:48:25.197555152 +0000 UTC m=+39.767094410"
Sep 10 14:48:28.589017 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:28.588970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:28.592686 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:28.592663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c065e49-1594-4143-8c64-d2894da542a5-metrics-tls\") pod \"dns-default-v46nj\" (UID: \"0c065e49-1594-4143-8c64-d2894da542a5\") " pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:28.598554 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:28.598535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:28.894208 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:28.894175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v46nj"]
Sep 10 14:48:28.897404 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:28.897377    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c065e49_1594_4143_8c64_d2894da542a5.slice/crio-5cbcbf03b10bc8bc1741ea685f139f9c303b617091b81f573294fe83145471d9 WatchSource:0}: Error finding container 5cbcbf03b10bc8bc1741ea685f139f9c303b617091b81f573294fe83145471d9: Status 404 returned error can't find the container with id 5cbcbf03b10bc8bc1741ea685f139f9c303b617091b81f573294fe83145471d9
Sep 10 14:48:29.192546 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:29.192460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v46nj" event={"ID":"0c065e49-1594-4143-8c64-d2894da542a5","Type":"ContainerStarted","Data":"5cbcbf03b10bc8bc1741ea685f139f9c303b617091b81f573294fe83145471d9"}
Sep 10 14:48:32.202135 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:32.202105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v46nj" event={"ID":"0c065e49-1594-4143-8c64-d2894da542a5","Type":"ContainerStarted","Data":"d94979581820b67f25057c9420c9890ac01e0f619195464386451b1e2a8b616e"}
Sep 10 14:48:32.202517 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:32.202157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v46nj" event={"ID":"0c065e49-1594-4143-8c64-d2894da542a5","Type":"ContainerStarted","Data":"70f0f1d40a64e5a07c4881af6be853198871cbcc7c9b768a496a58b0a54e7320"}
Sep 10 14:48:32.202517 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:32.202240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:32.219597 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:32.219552    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v46nj" podStartSLOduration=9.833342948 podStartE2EDuration="12.219538841s" totalImagesPullingTime="2.386195893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:32.218391462 +0000 UTC m=+46.787930736" watchObservedRunningTime="2026-09-10 14:48:32.219538841 +0000 UTC m=+46.789078098"
Sep 10 14:48:40.283506 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.283471    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"]
Sep 10 14:48:40.286434 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.286417    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-674fd9cf96-x9dtk"]
Sep 10 14:48:40.286557 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.286540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.289431 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.289406    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4clbj"]
Sep 10 14:48:40.289535 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.289496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.290061 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.290036    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 14:48:40.290170 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.290094    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 14:48:40.290170 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.290113    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 14:48:40.290170 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.290115    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 14:48:40.290322 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.290218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ms68p\""
Sep 10 14:48:40.292194 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.292173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.292878 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.292860    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 14:48:40.293132 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.293116    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 14:48:40.293278 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.293263    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 14:48:40.294057 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.294038    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 14:48:40.294207 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.294187    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 14:48:40.294281 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.294266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pzt6r\""
Sep 10 14:48:40.294515 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.294500    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 14:48:40.294576 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.294544    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 14:48:40.295202 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.295185    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7mjp9\""
Sep 10 14:48:40.296593 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.296573    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 14:48:40.296991 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.296976    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 14:48:40.297777 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.297615    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 14:48:40.301864 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.301843    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 14:48:40.305551 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.305531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674fd9cf96-x9dtk"]
Sep 10 14:48:40.306313 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.306294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"]
Sep 10 14:48:40.308957 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.308931    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4clbj"]
Sep 10 14:48:40.368281 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c05fd50-faf9-4283-b529-529d820501a4-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.368281 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368281    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c05fd50-faf9-4283-b529-529d820501a4-snapshots\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-stats-auth\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-default-certificate\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj8p\" (UniqueName: \"kubernetes.io/projected/73b9c56f-b47e-434e-b60a-32c8dc4946cc-kube-api-access-nsj8p\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c05fd50-faf9-4283-b529-529d820501a4-tmp\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmsr\" (UniqueName: \"kubernetes.io/projected/9c05fd50-faf9-4283-b529-529d820501a4-kube-api-access-kzmsr\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368465    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8tc\" (UniqueName: \"kubernetes.io/projected/014b8cd8-b074-48cf-8c18-2a325258fa41-kube-api-access-4h8tc\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73b9c56f-b47e-434e-b60a-32c8dc4946cc-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.368511 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-metrics-certs\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.368836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73b9c56f-b47e-434e-b60a-32c8dc4946cc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.368836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/014b8cd8-b074-48cf-8c18-2a325258fa41-service-ca-bundle\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.368836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c05fd50-faf9-4283-b529-529d820501a4-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.368836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.368615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c05fd50-faf9-4283-b529-529d820501a4-serving-cert\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.380883 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.380856    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-8dxmr"]
Sep 10 14:48:40.383782 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.383768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-8dxmr"
Sep 10 14:48:40.386157 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.386135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 14:48:40.386257 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.386235    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 14:48:40.386372 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.386358    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gkjh7\""
Sep 10 14:48:40.388764 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.388738    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bc9654f77-hdbls"]
Sep 10 14:48:40.391385 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.391363    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.393704 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.393685    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 14:48:40.393818 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.393704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 14:48:40.393818 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.393776    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 14:48:40.393955 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.393824    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgzsd\""
Sep 10 14:48:40.398740 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.398719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-8dxmr"]
Sep 10 14:48:40.403736 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.403717    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 14:48:40.413729 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.413704    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc9654f77-hdbls"]
Sep 10 14:48:40.469050 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76338a58-7e63-498b-bee4-c5d25aa8d878-image-registry-private-configuration\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469050 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-registry-tls\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c05fd50-faf9-4283-b529-529d820501a4-snapshots\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-stats-auth\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c05fd50-faf9-4283-b529-529d820501a4-tmp\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76338a58-7e63-498b-bee4-c5d25aa8d878-ca-trust-extracted\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8tc\" (UniqueName: \"kubernetes.io/projected/014b8cd8-b074-48cf-8c18-2a325258fa41-kube-api-access-4h8tc\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73b9c56f-b47e-434e-b60a-32c8dc4946cc-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-metrics-certs\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73b9c56f-b47e-434e-b60a-32c8dc4946cc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.469300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76338a58-7e63-498b-bee4-c5d25aa8d878-trusted-ca\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-bound-sa-token\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c05fd50-faf9-4283-b529-529d820501a4-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsr4\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-kube-api-access-qrsr4\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76338a58-7e63-498b-bee4-c5d25aa8d878-registry-certificates\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xq9\" (UniqueName: \"kubernetes.io/projected/b46acd3a-1ac6-4390-8297-96d7c384c7e4-kube-api-access-29xq9\") pod \"downloads-85d98c998f-8dxmr\" (UID: \"b46acd3a-1ac6-4390-8297-96d7c384c7e4\") " pod="openshift-console/downloads-85d98c998f-8dxmr"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-default-certificate\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj8p\" (UniqueName: \"kubernetes.io/projected/73b9c56f-b47e-434e-b60a-32c8dc4946cc-kube-api-access-nsj8p\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmsr\" (UniqueName: \"kubernetes.io/projected/9c05fd50-faf9-4283-b529-529d820501a4-kube-api-access-kzmsr\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/014b8cd8-b074-48cf-8c18-2a325258fa41-service-ca-bundle\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c05fd50-faf9-4283-b529-529d820501a4-tmp\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c05fd50-faf9-4283-b529-529d820501a4-serving-cert\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76338a58-7e63-498b-bee4-c5d25aa8d878-installation-pull-secrets\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.469822 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c05fd50-faf9-4283-b529-529d820501a4-snapshots\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.470488 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.469832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c05fd50-faf9-4283-b529-529d820501a4-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.470488 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.470177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/73b9c56f-b47e-434e-b60a-32c8dc4946cc-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.470597 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.470536    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c05fd50-faf9-4283-b529-529d820501a4-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.471156 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.471117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/014b8cd8-b074-48cf-8c18-2a325258fa41-service-ca-bundle\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.471320 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.471300    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c05fd50-faf9-4283-b529-529d820501a4-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.472175 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.472157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-stats-auth\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.472259 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.472229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-default-certificate\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.472603 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.472581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/73b9c56f-b47e-434e-b60a-32c8dc4946cc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.472783 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.472766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c05fd50-faf9-4283-b529-529d820501a4-serving-cert\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.473256 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.473231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014b8cd8-b074-48cf-8c18-2a325258fa41-metrics-certs\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.477465 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.477439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8tc\" (UniqueName: \"kubernetes.io/projected/014b8cd8-b074-48cf-8c18-2a325258fa41-kube-api-access-4h8tc\") pod \"router-default-674fd9cf96-x9dtk\" (UID: \"014b8cd8-b074-48cf-8c18-2a325258fa41\") " pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.478735 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.478715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmsr\" (UniqueName: \"kubernetes.io/projected/9c05fd50-faf9-4283-b529-529d820501a4-kube-api-access-kzmsr\") pod \"insights-operator-76bcd6ccf5-4clbj\" (UID: \"9c05fd50-faf9-4283-b529-529d820501a4\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.482851 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.482835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj8p\" (UniqueName: \"kubernetes.io/projected/73b9c56f-b47e-434e-b60a-32c8dc4946cc-kube-api-access-nsj8p\") pod \"cluster-monitoring-operator-6cdcc96965-8tz98\" (UID: \"73b9c56f-b47e-434e-b60a-32c8dc4946cc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.570354 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76338a58-7e63-498b-bee4-c5d25aa8d878-ca-trust-extracted\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570354 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76338a58-7e63-498b-bee4-c5d25aa8d878-trusted-ca\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570354 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-bound-sa-token\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570354 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsr4\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-kube-api-access-qrsr4\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570749 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76338a58-7e63-498b-bee4-c5d25aa8d878-registry-certificates\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570749 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xq9\" (UniqueName: \"kubernetes.io/projected/b46acd3a-1ac6-4390-8297-96d7c384c7e4-kube-api-access-29xq9\") pod \"downloads-85d98c998f-8dxmr\" (UID: \"b46acd3a-1ac6-4390-8297-96d7c384c7e4\") " pod="openshift-console/downloads-85d98c998f-8dxmr"
Sep 10 14:48:40.570749 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76338a58-7e63-498b-bee4-c5d25aa8d878-installation-pull-secrets\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570749 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76338a58-7e63-498b-bee4-c5d25aa8d878-image-registry-private-configuration\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570749 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76338a58-7e63-498b-bee4-c5d25aa8d878-ca-trust-extracted\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.570749 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.570716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-registry-tls\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.571303 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.571272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76338a58-7e63-498b-bee4-c5d25aa8d878-registry-certificates\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.572034 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.572010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76338a58-7e63-498b-bee4-c5d25aa8d878-trusted-ca\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.573000 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.572972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76338a58-7e63-498b-bee4-c5d25aa8d878-installation-pull-secrets\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.573073 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.573057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76338a58-7e63-498b-bee4-c5d25aa8d878-image-registry-private-configuration\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.573267 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.573244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-registry-tls\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.580961 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.580933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsr4\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-kube-api-access-qrsr4\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.581051 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.580980    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76338a58-7e63-498b-bee4-c5d25aa8d878-bound-sa-token\") pod \"image-registry-5bc9654f77-hdbls\" (UID: \"76338a58-7e63-498b-bee4-c5d25aa8d878\") " pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.581463 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.581447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xq9\" (UniqueName: \"kubernetes.io/projected/b46acd3a-1ac6-4390-8297-96d7c384c7e4-kube-api-access-29xq9\") pod \"downloads-85d98c998f-8dxmr\" (UID: \"b46acd3a-1ac6-4390-8297-96d7c384c7e4\") " pod="openshift-console/downloads-85d98c998f-8dxmr"
Sep 10 14:48:40.597258 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.597240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"
Sep 10 14:48:40.604401 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.604380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:40.612212 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.612190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj"
Sep 10 14:48:40.692154 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.692121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-8dxmr"
Sep 10 14:48:40.702360 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.702290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:40.868947 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.868906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98"]
Sep 10 14:48:40.872282 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:40.872207    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b9c56f_b47e_434e_b60a_32c8dc4946cc.slice/crio-afcc8f777a7e6d83198dd4592e6ff9d6a3bb6dac28629a1b5498d2f0663b3947 WatchSource:0}: Error finding container afcc8f777a7e6d83198dd4592e6ff9d6a3bb6dac28629a1b5498d2f0663b3947: Status 404 returned error can't find the container with id afcc8f777a7e6d83198dd4592e6ff9d6a3bb6dac28629a1b5498d2f0663b3947
Sep 10 14:48:40.876770 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.876731    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674fd9cf96-x9dtk"]
Sep 10 14:48:40.879465 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:40.879438    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014b8cd8_b074_48cf_8c18_2a325258fa41.slice/crio-55f06c7b952a386636c37a510854039000fdc65f34f78116a70a314b69cfe3a4 WatchSource:0}: Error finding container 55f06c7b952a386636c37a510854039000fdc65f34f78116a70a314b69cfe3a4: Status 404 returned error can't find the container with id 55f06c7b952a386636c37a510854039000fdc65f34f78116a70a314b69cfe3a4
Sep 10 14:48:40.898706 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.898674    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4clbj"]
Sep 10 14:48:40.906490 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:40.906467    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c05fd50_faf9_4283_b529_529d820501a4.slice/crio-5a72038f6a11dd197df53989dfc3609c06f5ceeb2bcdc3a907b2a2599627cd78 WatchSource:0}: Error finding container 5a72038f6a11dd197df53989dfc3609c06f5ceeb2bcdc3a907b2a2599627cd78: Status 404 returned error can't find the container with id 5a72038f6a11dd197df53989dfc3609c06f5ceeb2bcdc3a907b2a2599627cd78
Sep 10 14:48:40.979558 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.979408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-8dxmr"]
Sep 10 14:48:40.982261 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:40.981913    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46acd3a_1ac6_4390_8297_96d7c384c7e4.slice/crio-e479b9730cd4b398b6d584b1d715c88b788a2deccdf95ca88d92402f217a298c WatchSource:0}: Error finding container e479b9730cd4b398b6d584b1d715c88b788a2deccdf95ca88d92402f217a298c: Status 404 returned error can't find the container with id e479b9730cd4b398b6d584b1d715c88b788a2deccdf95ca88d92402f217a298c
Sep 10 14:48:40.983011 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:40.982990    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc9654f77-hdbls"]
Sep 10 14:48:40.985162 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:40.985137    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76338a58_7e63_498b_bee4_c5d25aa8d878.slice/crio-3e4b08e3755f495edaf0ee5b00bf183ae8530c523bc51c53294e17b634d9c227 WatchSource:0}: Error finding container 3e4b08e3755f495edaf0ee5b00bf183ae8530c523bc51c53294e17b634d9c227: Status 404 returned error can't find the container with id 3e4b08e3755f495edaf0ee5b00bf183ae8530c523bc51c53294e17b634d9c227
Sep 10 14:48:41.224571 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.224496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98" event={"ID":"73b9c56f-b47e-434e-b60a-32c8dc4946cc","Type":"ContainerStarted","Data":"afcc8f777a7e6d83198dd4592e6ff9d6a3bb6dac28629a1b5498d2f0663b3947"}
Sep 10 14:48:41.226025 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.226001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674fd9cf96-x9dtk" event={"ID":"014b8cd8-b074-48cf-8c18-2a325258fa41","Type":"ContainerStarted","Data":"1b387e2f7a5bc9fc153b80e4ca554982c874d3dfaa5e501152eade4069e4acf2"}
Sep 10 14:48:41.226157 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.226038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674fd9cf96-x9dtk" event={"ID":"014b8cd8-b074-48cf-8c18-2a325258fa41","Type":"ContainerStarted","Data":"55f06c7b952a386636c37a510854039000fdc65f34f78116a70a314b69cfe3a4"}
Sep 10 14:48:41.227623 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.227596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc9654f77-hdbls" event={"ID":"76338a58-7e63-498b-bee4-c5d25aa8d878","Type":"ContainerStarted","Data":"5c70061c29da620e3398f63fc0d566b538a85bd06c2c8c0565e50b8c40a2c923"}
Sep 10 14:48:41.227763 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.227666    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:48:41.227763 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.227688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc9654f77-hdbls" event={"ID":"76338a58-7e63-498b-bee4-c5d25aa8d878","Type":"ContainerStarted","Data":"3e4b08e3755f495edaf0ee5b00bf183ae8530c523bc51c53294e17b634d9c227"}
Sep 10 14:48:41.228881 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.228856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-8dxmr" event={"ID":"b46acd3a-1ac6-4390-8297-96d7c384c7e4","Type":"ContainerStarted","Data":"e479b9730cd4b398b6d584b1d715c88b788a2deccdf95ca88d92402f217a298c"}
Sep 10 14:48:41.230115 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.230095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj" event={"ID":"9c05fd50-faf9-4283-b529-529d820501a4","Type":"ContainerStarted","Data":"5a72038f6a11dd197df53989dfc3609c06f5ceeb2bcdc3a907b2a2599627cd78"}
Sep 10 14:48:41.242904 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.242862    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-674fd9cf96-x9dtk" podStartSLOduration=1.242846403 podStartE2EDuration="1.242846403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:41.242122746 +0000 UTC m=+55.811662005" watchObservedRunningTime="2026-09-10 14:48:41.242846403 +0000 UTC m=+55.812385662"
Sep 10 14:48:41.259492 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.259441    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bc9654f77-hdbls" podStartSLOduration=1.259427825 podStartE2EDuration="1.259427825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:41.258912105 +0000 UTC m=+55.828451362" watchObservedRunningTime="2026-09-10 14:48:41.259427825 +0000 UTC m=+55.828967080"
Sep 10 14:48:41.605510 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.605476    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:41.608809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:41.608784    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:42.208449 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:42.208106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v46nj"
Sep 10 14:48:42.236300 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:42.236270    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:42.237932 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:42.237745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-674fd9cf96-x9dtk"
Sep 10 14:48:44.245894 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.245846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj" event={"ID":"9c05fd50-faf9-4283-b529-529d820501a4","Type":"ContainerStarted","Data":"a21f5181710b367746d10c96238ea5ed066daa8594f50ddeed683c97d3566826"}
Sep 10 14:48:44.248565 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.248505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98" event={"ID":"73b9c56f-b47e-434e-b60a-32c8dc4946cc","Type":"ContainerStarted","Data":"499066f7674b5c7457da37ca863c5f2b39a9bb1fa541a3f3b23559654b4084ce"}
Sep 10 14:48:44.265890 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.265839    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-4clbj" podStartSLOduration=1.44490596 podStartE2EDuration="4.265819519s" totalImagesPullingTime="2.820913559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:44.265035875 +0000 UTC m=+58.834575133" watchObservedRunningTime="2026-09-10 14:48:44.265819519 +0000 UTC m=+58.835358778"
Sep 10 14:48:44.281957 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.281439    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8tz98" podStartSLOduration=1.4307665250000001 podStartE2EDuration="4.281417559s" totalImagesPullingTime="2.850651034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:44.280259116 +0000 UTC m=+58.849798375" watchObservedRunningTime="2026-09-10 14:48:44.281417559 +0000 UTC m=+58.850956818"
Sep 10 14:48:44.321379 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.321335    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7khtl"]
Sep 10 14:48:44.324873 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.324851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.327107 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.327079    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 14:48:44.327225 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.327151    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g76rw\""
Sep 10 14:48:44.327516 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.327488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 14:48:44.343821 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.343775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7khtl"]
Sep 10 14:48:44.404466 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.404431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6ee187-c103-44b3-80d1-3ce30747905c-data-volume\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.404636 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.404499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7qj\" (UniqueName: \"kubernetes.io/projected/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-api-access-bj7qj\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.404636 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.404559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.404636 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.404591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.404636 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.404617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b6ee187-c103-44b3-80d1-3ce30747905c-crio-socket\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.505731 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.505585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.505731 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.505659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.505731 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.505697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b6ee187-c103-44b3-80d1-3ce30747905c-crio-socket\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.506152 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:44.505759    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:44.506152 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.505815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6ee187-c103-44b3-80d1-3ce30747905c-data-volume\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.506152 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:44.505848    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:45.005828461 +0000 UTC m=+59.575367704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:44.506152 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.505904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b6ee187-c103-44b3-80d1-3ce30747905c-crio-socket\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.506152 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.505908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7qj\" (UniqueName: \"kubernetes.io/projected/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-api-access-bj7qj\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.506152 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:44.505773    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:48:44.506152 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:44.505994    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:45.005981469 +0000 UTC m=+59.575520737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : configmap "kube-rbac-proxy" not found
Sep 10 14:48:44.506519 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.506158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b6ee187-c103-44b3-80d1-3ce30747905c-data-volume\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:44.514028 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:44.514003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7qj\" (UniqueName: \"kubernetes.io/projected/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-api-access-bj7qj\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:45.009801 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:45.009762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:45.009801 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:45.009805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:45.009980 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:45.009919    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:45.010012 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:45.009992    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:46.009978022 +0000 UTC m=+60.579517262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:45.010049 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:45.009919    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:48:45.010079 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:45.010063    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:46.010050771 +0000 UTC m=+60.579590024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : configmap "kube-rbac-proxy" not found
Sep 10 14:48:45.157328 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:45.157296    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxl49"
Sep 10 14:48:45.728684 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:45.728633    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v46nj_0c065e49-1594-4143-8c64-d2894da542a5/dns/0.log"
Sep 10 14:48:45.913111 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:45.913085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v46nj_0c065e49-1594-4143-8c64-d2894da542a5/kube-rbac-proxy/0.log"
Sep 10 14:48:46.019600 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:46.019514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:46.019600 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:46.019551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:46.019848 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:46.019634    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:48:46.019848 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:46.019690    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:46.019848 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:46.019732    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:48.019713268 +0000 UTC m=+62.589252523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : configmap "kube-rbac-proxy" not found
Sep 10 14:48:46.019848 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:46.019747    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:48.019741239 +0000 UTC m=+62.589280478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:46.513397 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:46.513369    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xzqgt_ae64e0a9-6159-40ff-879c-f7ff9ce3097e/dns-node-resolver/0.log"
Sep 10 14:48:46.713565 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:46.713530    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bc9654f77-hdbls_76338a58-7e63-498b-bee4-c5d25aa8d878/registry/0.log"
Sep 10 14:48:47.301705 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.301670    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"]
Sep 10 14:48:47.306854 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.306833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.309136 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.308943    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 14:48:47.309136 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.308978    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 14:48:47.309136 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.308981    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tskmc\""
Sep 10 14:48:47.309326 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.309260    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 14:48:47.313292 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.313237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"]
Sep 10 14:48:47.316346 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.316325    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8d9nt_9f8134df-e311-49b1-bbeb-59171cd288eb/node-ca/0.log"
Sep 10 14:48:47.333315 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.333294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.333438 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.333328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.333507 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.333447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r87f\" (UniqueName: \"kubernetes.io/projected/5e6ec021-6bce-4edd-8161-76d0da70f01a-kube-api-access-6r87f\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.333507 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.333497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ec021-6bce-4edd-8161-76d0da70f01a-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.434692 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.434653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.434848 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.434719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.434848 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:47.434754    2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 14:48:47.434848 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:47.434822    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-tls podName:5e6ec021-6bce-4edd-8161-76d0da70f01a nodeName:}" failed. No retries permitted until 2026-09-10 14:48:47.934805781 +0000 UTC m=+62.504345030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-wl9j6" (UID: "5e6ec021-6bce-4edd-8161-76d0da70f01a") : secret "prometheus-operator-tls" not found
Sep 10 14:48:47.434848 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.434840    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r87f\" (UniqueName: \"kubernetes.io/projected/5e6ec021-6bce-4edd-8161-76d0da70f01a-kube-api-access-6r87f\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.435074 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.434874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ec021-6bce-4edd-8161-76d0da70f01a-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.435499 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.435476    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e6ec021-6bce-4edd-8161-76d0da70f01a-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.437377 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.437357    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.444983 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.444955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r87f\" (UniqueName: \"kubernetes.io/projected/5e6ec021-6bce-4edd-8161-76d0da70f01a-kube-api-access-6r87f\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.913312 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.913276    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674fd9cf96-x9dtk_014b8cd8-b074-48cf-8c18-2a325258fa41/router/0.log"
Sep 10 14:48:47.938301 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.938267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:47.941086 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:47.941058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e6ec021-6bce-4edd-8161-76d0da70f01a-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wl9j6\" (UID: \"5e6ec021-6bce-4edd-8161-76d0da70f01a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:48.039305 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:48.039272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:48.039493 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:48.039320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:48.039493 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:48.039443    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:48.039493 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:48.039485    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:48:48.039674 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:48.039535    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:52.039515564 +0000 UTC m=+66.609054817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:48:48.039674 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:48.039552    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm podName:0b6ee187-c103-44b3-80d1-3ce30747905c nodeName:}" failed. No retries permitted until 2026-09-10 14:48:52.039545048 +0000 UTC m=+66.609084283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7khtl" (UID: "0b6ee187-c103-44b3-80d1-3ce30747905c") : configmap "kube-rbac-proxy" not found
Sep 10 14:48:48.217975 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:48.217895    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"
Sep 10 14:48:48.496253 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:48.496179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wl9j6"]
Sep 10 14:48:48.499673 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:48.499629    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6ec021_6bce_4edd_8161_76d0da70f01a.slice/crio-247e96e1be6da550181b6e2f496f4a46055133325b2965f4bcc3d437b784699f WatchSource:0}: Error finding container 247e96e1be6da550181b6e2f496f4a46055133325b2965f4bcc3d437b784699f: Status 404 returned error can't find the container with id 247e96e1be6da550181b6e2f496f4a46055133325b2965f4bcc3d437b784699f
Sep 10 14:48:48.512725 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:48.512611    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ttnzd_9a01ff9d-c190-49e0-8196-405a2df80685/serve-healthcheck-canary/0.log"
Sep 10 14:48:49.269128 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:49.269082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6" event={"ID":"5e6ec021-6bce-4edd-8161-76d0da70f01a","Type":"ContainerStarted","Data":"247e96e1be6da550181b6e2f496f4a46055133325b2965f4bcc3d437b784699f"}
Sep 10 14:48:51.280593 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.280557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6" event={"ID":"5e6ec021-6bce-4edd-8161-76d0da70f01a","Type":"ContainerStarted","Data":"b06d79a6a6b277e05f30a719d4dc7718228c53d5021215ba30a7ee793a68e3ec"}
Sep 10 14:48:51.281039 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.280611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6" event={"ID":"5e6ec021-6bce-4edd-8161-76d0da70f01a","Type":"ContainerStarted","Data":"ba4ed690417153b8d142aa77a74760bd98045592edf22602b9f288dc477edeac"}
Sep 10 14:48:51.299220 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.299166    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-wl9j6" podStartSLOduration=2.517053141 podStartE2EDuration="4.299150037s" totalImagesPullingTime="1.782096896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:51.297941013 +0000 UTC m=+65.867480297" watchObservedRunningTime="2026-09-10 14:48:51.299150037 +0000 UTC m=+65.868689294"
Sep 10 14:48:51.673099 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.673059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:51.675342 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.675314    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:48:51.686588 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.686562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0016d121-63c2-4d58-8110-81f9d47e1a64-metrics-certs\") pod \"network-metrics-daemon-2vr4h\" (UID: \"0016d121-63c2-4d58-8110-81f9d47e1a64\") " pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:51.774063 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.774025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:51.774251 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.774076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:51.776443 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.776407    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:48:51.776577 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.776466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:48:51.786943 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.786921    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:48:51.787753 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.787722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fbf0051-5f2c-4ef2-af7b-c00c1b52241d-original-pull-secret\") pod \"global-pull-secret-syncer-vbtcs\" (UID: \"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d\") " pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:51.797986 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.797963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqrq\" (UniqueName: \"kubernetes.io/projected/4dc9c946-3156-4721-b715-4b0045879e99-kube-api-access-6bqrq\") pod \"network-check-target-mn2b9\" (UID: \"4dc9c946-3156-4721-b715-4b0045879e99\") " pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:51.881094 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.881058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbtcs"
Sep 10 14:48:51.889344 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.889320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sc2gb\""
Sep 10 14:48:51.892850 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.892831    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fr4wz\""
Sep 10 14:48:51.897597 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.897574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:48:51.901212 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:51.901196    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vr4h"
Sep 10 14:48:52.077219 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:52.077123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:52.077219 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:52.077182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:52.077904 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:52.077877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b6ee187-c103-44b3-80d1-3ce30747905c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:52.079658 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:52.079622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b6ee187-c103-44b3-80d1-3ce30747905c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7khtl\" (UID: \"0b6ee187-c103-44b3-80d1-3ce30747905c\") " pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:52.141634 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:52.141581    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g76rw\""
Sep 10 14:48:52.149551 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:52.149525    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7khtl"
Sep 10 14:48:53.634271 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.634237    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"]
Sep 10 14:48:53.637851 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.637812    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.640005 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.639968    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 14:48:53.640126 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.639972    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 14:48:53.640689 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.640669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kf9xj\""
Sep 10 14:48:53.653371 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.649674    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"]
Sep 10 14:48:53.684048 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.684011    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6bgbm"]
Sep 10 14:48:53.687225 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.687204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.689655 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.689615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 14:48:53.689769 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.689632    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c5gfh\""
Sep 10 14:48:53.689769 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.689696    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 14:48:53.689987 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.689961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 14:48:53.691428 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.691396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.691537 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.691458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.691592 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.691557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmr85\" (UniqueName: \"kubernetes.io/projected/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-kube-api-access-nmr85\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.691670 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.691617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.792213 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.792213 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-wtmp\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792452 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.792452 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-tls\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792452 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.792452 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792681 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885d3e75-5483-4379-a070-d05ec15fb918-metrics-client-ca\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792681 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-root\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792681 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-sys\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792681 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792887 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk54c\" (UniqueName: \"kubernetes.io/projected/885d3e75-5483-4379-a070-d05ec15fb918-kube-api-access-nk54c\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.792887 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmr85\" (UniqueName: \"kubernetes.io/projected/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-kube-api-access-nmr85\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.792887 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.792801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-textfile\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.793207 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.793180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.795100 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.795081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.795197 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.795113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.800027 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.800009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmr85\" (UniqueName: \"kubernetes.io/projected/0ea71623-bc6b-4d11-b0b5-16e4128abcc6-kube-api-access-nmr85\") pod \"openshift-state-metrics-657fd6c54-nn6sc\" (UID: \"0ea71623-bc6b-4d11-b0b5-16e4128abcc6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:53.893769 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-wtmp\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893769 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-tls\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885d3e75-5483-4379-a070-d05ec15fb918-metrics-client-ca\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-root\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-sys\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-wtmp\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:53.893887    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk54c\" (UniqueName: \"kubernetes.io/projected/885d3e75-5483-4379-a070-d05ec15fb918-kube-api-access-nk54c\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.893960 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:48:53.893953    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-tls podName:885d3e75-5483-4379-a070-d05ec15fb918 nodeName:}" failed. No retries permitted until 2026-09-10 14:48:54.393932383 +0000 UTC m=+68.963471637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-tls") pod "node-exporter-6bgbm" (UID: "885d3e75-5483-4379-a070-d05ec15fb918") : secret "node-exporter-tls" not found
Sep 10 14:48:53.894374 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.893995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-textfile\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.894374 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.894164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-root\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.894374 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.894271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885d3e75-5483-4379-a070-d05ec15fb918-sys\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.894520 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.894489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.894520 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.894508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-textfile\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.894664 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.894626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885d3e75-5483-4379-a070-d05ec15fb918-metrics-client-ca\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.896708 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.896685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.901699 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.901630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk54c\" (UniqueName: \"kubernetes.io/projected/885d3e75-5483-4379-a070-d05ec15fb918-kube-api-access-nk54c\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:53.949450 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:53.949418    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"
Sep 10 14:48:54.398823 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.398791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-tls\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:54.401759 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.401725    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/885d3e75-5483-4379-a070-d05ec15fb918-node-exporter-tls\") pod \"node-exporter-6bgbm\" (UID: \"885d3e75-5483-4379-a070-d05ec15fb918\") " pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:54.598477 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.598444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6bgbm"
Sep 10 14:48:54.913439 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.913403    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7995c98d7b-jsl8q"]
Sep 10 14:48:54.916940 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.916910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:54.919211 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.919185    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 14:48:54.919339 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.919194    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 14:48:54.919896 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.919872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 14:48:54.920016 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.919924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 14:48:54.920215 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.920194    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h7btv\""
Sep 10 14:48:54.920288 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.920249    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 14:48:54.924824 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.924798    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 14:48:54.926705 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:54.926686    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7995c98d7b-jsl8q"]
Sep 10 14:48:55.004654 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.004617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-oauth-serving-cert\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.004838 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.004687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-trusted-ca-bundle\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.004838 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.004725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-serving-cert\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.004838 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.004817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-oauth-config\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.005001 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.004878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w29l\" (UniqueName: \"kubernetes.io/projected/82b83f2b-6a77-49dc-974f-20b02c22c495-kube-api-access-2w29l\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.005001 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.004972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-console-config\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.005103 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.005017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-service-ca\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.105454 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.105419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-trusted-ca-bundle\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.105663 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.105462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-serving-cert\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.105663 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.105510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-oauth-config\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.105663 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.105531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w29l\" (UniqueName: \"kubernetes.io/projected/82b83f2b-6a77-49dc-974f-20b02c22c495-kube-api-access-2w29l\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.105663 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.105591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-console-config\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.105663 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.105615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-service-ca\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.105886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.105673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-oauth-serving-cert\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.106377 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.106346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-oauth-serving-cert\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.106518 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.106419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-trusted-ca-bundle\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.106718 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.106695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-service-ca\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.106817 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.106728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-console-config\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.108541 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.108505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-serving-cert\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.108541 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.108511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-oauth-config\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.113795 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.113770    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w29l\" (UniqueName: \"kubernetes.io/projected/82b83f2b-6a77-49dc-974f-20b02c22c495-kube-api-access-2w29l\") pod \"console-7995c98d7b-jsl8q\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") " pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:55.233005 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:55.232729    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:48:57.953718 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.953680    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-559fb774d7-fkhg6"]
Sep 10 14:48:57.960067 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.960045    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:57.962523 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.962499    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 14:48:57.963399 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.963360    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 14:48:57.963399 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.963392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 14:48:57.963590 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.963448    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xkgm2\""
Sep 10 14:48:57.963590 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.963485    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 14:48:57.963744 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.963595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fdmppiv86m7k6\""
Sep 10 14:48:57.968042 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:57.968023    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-559fb774d7-fkhg6"]
Sep 10 14:48:58.031699 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.031664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-client-ca-bundle\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.031863 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.031714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249mq\" (UniqueName: \"kubernetes.io/projected/7685a9a9-0243-46cb-af68-c7f43466138a-kube-api-access-249mq\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.031863 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.031746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-secret-metrics-server-tls\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.031863 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.031793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7685a9a9-0243-46cb-af68-c7f43466138a-audit-log\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.031863 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.031828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7685a9a9-0243-46cb-af68-c7f43466138a-metrics-server-audit-profiles\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.032067 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.031921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-secret-metrics-server-client-certs\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.032067 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.031989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7685a9a9-0243-46cb-af68-c7f43466138a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.057062 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:58.057026    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885d3e75_5483_4379_a070_d05ec15fb918.slice/crio-23883d7f3cb782fd60284a1b327306f55142ab5e06625d952bd6bfc821c7c11d WatchSource:0}: Error finding container 23883d7f3cb782fd60284a1b327306f55142ab5e06625d952bd6bfc821c7c11d: Status 404 returned error can't find the container with id 23883d7f3cb782fd60284a1b327306f55142ab5e06625d952bd6bfc821c7c11d
Sep 10 14:48:58.142347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.142044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-secret-metrics-server-client-certs\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.142347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.142123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7685a9a9-0243-46cb-af68-c7f43466138a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.142347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.142165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-client-ca-bundle\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.142347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.142181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249mq\" (UniqueName: \"kubernetes.io/projected/7685a9a9-0243-46cb-af68-c7f43466138a-kube-api-access-249mq\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.142347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.142204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-secret-metrics-server-tls\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.142347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.142229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7685a9a9-0243-46cb-af68-c7f43466138a-audit-log\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.142347 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.142254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7685a9a9-0243-46cb-af68-c7f43466138a-metrics-server-audit-profiles\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.144443 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.144391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7685a9a9-0243-46cb-af68-c7f43466138a-metrics-server-audit-profiles\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.145432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.145059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7685a9a9-0243-46cb-af68-c7f43466138a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.145432 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.145303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7685a9a9-0243-46cb-af68-c7f43466138a-audit-log\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.147316 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.146756    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-secret-metrics-server-tls\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.147316 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.147254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-secret-metrics-server-client-certs\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.148876 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.148841    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7685a9a9-0243-46cb-af68-c7f43466138a-client-ca-bundle\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.151694 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.151674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-249mq\" (UniqueName: \"kubernetes.io/projected/7685a9a9-0243-46cb-af68-c7f43466138a-kube-api-access-249mq\") pod \"metrics-server-559fb774d7-fkhg6\" (UID: \"7685a9a9-0243-46cb-af68-c7f43466138a\") " pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.271779 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.271747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:48:58.307679 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.307624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-8dxmr" event={"ID":"b46acd3a-1ac6-4390-8297-96d7c384c7e4","Type":"ContainerStarted","Data":"b7df26d4be10c8f82f2009bdd6241cdd5147e3932a6cb08c41c3d1430eac38b4"}
Sep 10 14:48:58.307863 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.307839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-8dxmr"
Sep 10 14:48:58.309112 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.309087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bgbm" event={"ID":"885d3e75-5483-4379-a070-d05ec15fb918","Type":"ContainerStarted","Data":"23883d7f3cb782fd60284a1b327306f55142ab5e06625d952bd6bfc821c7c11d"}
Sep 10 14:48:58.309949 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.309628    2565 patch_prober.go:28] interesting pod/downloads-85d98c998f-8dxmr container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused" start-of-body=
Sep 10 14:48:58.309949 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.309706    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-8dxmr" podUID="b46acd3a-1ac6-4390-8297-96d7c384c7e4" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused"
Sep 10 14:48:58.325862 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.325730    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-8dxmr" podStartSLOduration=1.13629703 podStartE2EDuration="18.325709232s" totalImagesPullingTime="17.189412202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:58.32490316 +0000 UTC m=+72.894442420" watchObservedRunningTime="2026-09-10 14:48:58.325709232 +0000 UTC m=+72.895248493"
Sep 10 14:48:58.335925 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.335887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc"]
Sep 10 14:48:58.338712 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:58.338624    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea71623_bc6b_4d11_b0b5_16e4128abcc6.slice/crio-261ddb0d45ddf23626398254d0239e8f03f24b72764b242f6460bddc85c45639 WatchSource:0}: Error finding container 261ddb0d45ddf23626398254d0239e8f03f24b72764b242f6460bddc85c45639: Status 404 returned error can't find the container with id 261ddb0d45ddf23626398254d0239e8f03f24b72764b242f6460bddc85c45639
Sep 10 14:48:58.340064 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.340024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vr4h"]
Sep 10 14:48:58.343769 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:58.343736    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0016d121_63c2_4d58_8110_81f9d47e1a64.slice/crio-43b67feec9355696e324edae81413569dd89a8093e2c097d5660f4ec0e559452 WatchSource:0}: Error finding container 43b67feec9355696e324edae81413569dd89a8093e2c097d5660f4ec0e559452: Status 404 returned error can't find the container with id 43b67feec9355696e324edae81413569dd89a8093e2c097d5660f4ec0e559452
Sep 10 14:48:58.347346 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.347002    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7khtl"]
Sep 10 14:48:58.352202 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:58.352176    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6ee187_c103_44b3_80d1_3ce30747905c.slice/crio-b80471bc394329f38ae4ca9df6604ef70157fe3fa443176aa3a0005c70ccca1c WatchSource:0}: Error finding container b80471bc394329f38ae4ca9df6604ef70157fe3fa443176aa3a0005c70ccca1c: Status 404 returned error can't find the container with id b80471bc394329f38ae4ca9df6604ef70157fe3fa443176aa3a0005c70ccca1c
Sep 10 14:48:58.397688 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.396517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbtcs"]
Sep 10 14:48:58.403706 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.403298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7995c98d7b-jsl8q"]
Sep 10 14:48:58.408402 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.405559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mn2b9"]
Sep 10 14:48:58.413414 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:58.411764    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbf0051_5f2c_4ef2_af7b_c00c1b52241d.slice/crio-fd75ca37aedb8590c4e983db9c0df8832ea9a8280cd19012d2bcf1490b66947c WatchSource:0}: Error finding container fd75ca37aedb8590c4e983db9c0df8832ea9a8280cd19012d2bcf1490b66947c: Status 404 returned error can't find the container with id fd75ca37aedb8590c4e983db9c0df8832ea9a8280cd19012d2bcf1490b66947c
Sep 10 14:48:58.556363 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:58.555681    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-559fb774d7-fkhg6"]
Sep 10 14:48:58.558806 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:48:58.558776    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7685a9a9_0243_46cb_af68_c7f43466138a.slice/crio-7def84bc774434e723887169f675473aff341335ec40106a631c3c9abf8c6c18 WatchSource:0}: Error finding container 7def84bc774434e723887169f675473aff341335ec40106a631c3c9abf8c6c18: Status 404 returned error can't find the container with id 7def84bc774434e723887169f675473aff341335ec40106a631c3c9abf8c6c18
Sep 10 14:48:59.318635 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.318549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vr4h" event={"ID":"0016d121-63c2-4d58-8110-81f9d47e1a64","Type":"ContainerStarted","Data":"43b67feec9355696e324edae81413569dd89a8093e2c097d5660f4ec0e559452"}
Sep 10 14:48:59.321625 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.321547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6" event={"ID":"7685a9a9-0243-46cb-af68-c7f43466138a","Type":"ContainerStarted","Data":"7def84bc774434e723887169f675473aff341335ec40106a631c3c9abf8c6c18"}
Sep 10 14:48:59.324679 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.324635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vbtcs" event={"ID":"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d","Type":"ContainerStarted","Data":"fd75ca37aedb8590c4e983db9c0df8832ea9a8280cd19012d2bcf1490b66947c"}
Sep 10 14:48:59.329877 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.329845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7khtl" event={"ID":"0b6ee187-c103-44b3-80d1-3ce30747905c","Type":"ContainerStarted","Data":"b9a00f01af6b665920c97bbaccd2c8f362ca17976aaa0d1ae1866e05d2de5aa1"}
Sep 10 14:48:59.329976 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.329909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7khtl" event={"ID":"0b6ee187-c103-44b3-80d1-3ce30747905c","Type":"ContainerStarted","Data":"b80471bc394329f38ae4ca9df6604ef70157fe3fa443176aa3a0005c70ccca1c"}
Sep 10 14:48:59.331561 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.331492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mn2b9" event={"ID":"4dc9c946-3156-4721-b715-4b0045879e99","Type":"ContainerStarted","Data":"a8f92280dab4bfb64c5752e68511a337bf5a5aa290145f548ea43da8d25e0bfa"}
Sep 10 14:48:59.334546 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.334419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc" event={"ID":"0ea71623-bc6b-4d11-b0b5-16e4128abcc6","Type":"ContainerStarted","Data":"04fcf6da3894444f67fb8fe4fb0214b23bed1a0666f1ce029024cb5c12b91a11"}
Sep 10 14:48:59.334546 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.334471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc" event={"ID":"0ea71623-bc6b-4d11-b0b5-16e4128abcc6","Type":"ContainerStarted","Data":"a676ecb6eb89dc4584d2d79993b9f78da93c3eed0b1b0d49efe9ceeacb9e5888"}
Sep 10 14:48:59.334546 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.334499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc" event={"ID":"0ea71623-bc6b-4d11-b0b5-16e4128abcc6","Type":"ContainerStarted","Data":"261ddb0d45ddf23626398254d0239e8f03f24b72764b242f6460bddc85c45639"}
Sep 10 14:48:59.337555 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.337532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7995c98d7b-jsl8q" event={"ID":"82b83f2b-6a77-49dc-974f-20b02c22c495","Type":"ContainerStarted","Data":"810a379633fe80aef11e5c02dff4c98e13145c8039318901134c5410104eb97e"}
Sep 10 14:48:59.351462 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:48:59.351414    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-8dxmr"
Sep 10 14:49:01.350751 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.350667    2565 generic.go:356] "Generic (PLEG): container finished" podID="885d3e75-5483-4379-a070-d05ec15fb918" containerID="1f8fd765efc3f541422c54bd9f1cdc39934541a4e55c0bd30a5d2fd25e02d7a2" exitCode=0
Sep 10 14:49:01.351211 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.350764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bgbm" event={"ID":"885d3e75-5483-4379-a070-d05ec15fb918","Type":"ContainerDied","Data":"1f8fd765efc3f541422c54bd9f1cdc39934541a4e55c0bd30a5d2fd25e02d7a2"}
Sep 10 14:49:01.734127 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.734031    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7995c98d7b-jsl8q"]
Sep 10 14:49:01.761677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.760573    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6476cbccb4-lhsk6"]
Sep 10 14:49:01.787476 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.787426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6476cbccb4-lhsk6"]
Sep 10 14:49:01.787684 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.787566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.894677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.894257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-oauth-serving-cert\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.894677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.894310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxfl\" (UniqueName: \"kubernetes.io/projected/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-kube-api-access-dhxfl\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.894677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.894337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-oauth-config\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.894677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.894394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-trusted-ca-bundle\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.894677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.894438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-config\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.894677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.894513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-serving-cert\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.894677 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.894543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-service-ca\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.996448 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.995741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-trusted-ca-bundle\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.996448 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.995798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-config\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.996448 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.995858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-serving-cert\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.996448 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.995889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-service-ca\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.996448 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.995916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-oauth-serving-cert\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.996448 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.995942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxfl\" (UniqueName: \"kubernetes.io/projected/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-kube-api-access-dhxfl\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.996448 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.995967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-oauth-config\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.998670 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.996828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-config\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.998670 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.997284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-service-ca\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.998670 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.997710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-trusted-ca-bundle\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.998670 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.998226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-oauth-serving-cert\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:01.999469 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:01.999445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-oauth-config\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:02.001202 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:02.001175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-serving-cert\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:02.006036 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:02.005991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxfl\" (UniqueName: \"kubernetes.io/projected/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-kube-api-access-dhxfl\") pod \"console-6476cbccb4-lhsk6\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") " pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:02.102627 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:02.102374    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:02.242691 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:02.242658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bc9654f77-hdbls"
Sep 10 14:49:05.374232 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:05.374189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vr4h" event={"ID":"0016d121-63c2-4d58-8110-81f9d47e1a64","Type":"ContainerStarted","Data":"9517ef9626ead98b8b2375f5668ae905c91733d280bbb2720c65bb571c9bf733"}
Sep 10 14:49:08.327346 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:08.327318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6476cbccb4-lhsk6"]
Sep 10 14:49:08.341354 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:49:08.341319    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb382e9b0_1834_4dd9_8b4d_fc8a5037816b.slice/crio-442b9f3c368675460100ded71683b664226d5764c6233f9150260789e14adf36 WatchSource:0}: Error finding container 442b9f3c368675460100ded71683b664226d5764c6233f9150260789e14adf36: Status 404 returned error can't find the container with id 442b9f3c368675460100ded71683b664226d5764c6233f9150260789e14adf36
Sep 10 14:49:08.390398 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:08.390362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6476cbccb4-lhsk6" event={"ID":"b382e9b0-1834-4dd9-8b4d-fc8a5037816b","Type":"ContainerStarted","Data":"442b9f3c368675460100ded71683b664226d5764c6233f9150260789e14adf36"}
Sep 10 14:49:08.394056 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:08.393361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bgbm" event={"ID":"885d3e75-5483-4379-a070-d05ec15fb918","Type":"ContainerStarted","Data":"adaa94ef0a3947581bc17e3fe6ca2b80f1e2be9adfa26fd5832bd83a93fa8d0d"}
Sep 10 14:49:08.396445 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:08.396421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7khtl" event={"ID":"0b6ee187-c103-44b3-80d1-3ce30747905c","Type":"ContainerStarted","Data":"f2f1c8c7661e20eeb268709df1830d7a7fde0726cec21fa73f4bcb6b97c0ae2f"}
Sep 10 14:49:08.398525 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:08.398502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vr4h" event={"ID":"0016d121-63c2-4d58-8110-81f9d47e1a64","Type":"ContainerStarted","Data":"15c46cab61d1b3e3db7786b4d3af98c15e4f79f0c22a276f787afe8437b2e275"}
Sep 10 14:49:08.426023 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:08.425697    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2vr4h" podStartSLOduration=80.215878068 podStartE2EDuration="1m22.425618082s" totalImagesPullingTime="2.209740014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:08.424287384 +0000 UTC m=+82.993826644" watchObservedRunningTime="2026-09-10 14:49:08.425618082 +0000 UTC m=+82.995157342"
Sep 10 14:49:09.404785 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.404752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bgbm" event={"ID":"885d3e75-5483-4379-a070-d05ec15fb918","Type":"ContainerStarted","Data":"1ec00f4447ea8723b695eafb381d6c3b366ca8302bd7cf747700c5c9012eec9d"}
Sep 10 14:49:09.406289 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.406254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mn2b9" event={"ID":"4dc9c946-3156-4721-b715-4b0045879e99","Type":"ContainerStarted","Data":"d826788edf8e6b90b82af645985db86bd2c411d555f1bd89b87f25987d36105c"}
Sep 10 14:49:09.406420 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.406339    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:49:09.408302 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.408274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc" event={"ID":"0ea71623-bc6b-4d11-b0b5-16e4128abcc6","Type":"ContainerStarted","Data":"1d6c6aa60e2827a5eecbd279cfb119d431d8987b5a4b12a9dba5aee9456192cf"}
Sep 10 14:49:09.409858 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.409814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7995c98d7b-jsl8q" event={"ID":"82b83f2b-6a77-49dc-974f-20b02c22c495","Type":"ContainerStarted","Data":"d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e"}
Sep 10 14:49:09.411394 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.411369    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6" event={"ID":"7685a9a9-0243-46cb-af68-c7f43466138a","Type":"ContainerStarted","Data":"2e617520c483a83edadae87f6a639b3a7f79759bd24c77eb55333eb79399810d"}
Sep 10 14:49:09.412818 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.412790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vbtcs" event={"ID":"2fbf0051-5f2c-4ef2-af7b-c00c1b52241d","Type":"ContainerStarted","Data":"00341be90e104062ba8c8bc26b8bade36a0786eaa2766908a078c688d59071ea"}
Sep 10 14:49:09.414239 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.414216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6476cbccb4-lhsk6" event={"ID":"b382e9b0-1834-4dd9-8b4d-fc8a5037816b","Type":"ContainerStarted","Data":"de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00"}
Sep 10 14:49:09.424171 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.424118    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6bgbm" podStartSLOduration=13.862628943 podStartE2EDuration="16.42408207s" totalImagesPullingTime="2.453127144s" totalInitContainerRuntime="108.325983ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:09.42243982 +0000 UTC m=+83.991979078" watchObservedRunningTime="2026-09-10 14:49:09.42408207 +0000 UTC m=+83.993621327"
Sep 10 14:49:09.436190 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.436151    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vbtcs" podStartSLOduration=73.582561742 podStartE2EDuration="1m23.436139436s" totalImagesPullingTime="9.853577694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:09.435475346 +0000 UTC m=+84.005014604" watchObservedRunningTime="2026-09-10 14:49:09.436139436 +0000 UTC m=+84.005678694"
Sep 10 14:49:09.451312 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.451267    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-nn6sc" podStartSLOduration=6.697249498 podStartE2EDuration="16.451254853s" totalImagesPullingTime="9.754005355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:09.450744865 +0000 UTC m=+84.020284122" watchObservedRunningTime="2026-09-10 14:49:09.451254853 +0000 UTC m=+84.020794110"
Sep 10 14:49:09.470402 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.470353    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7995c98d7b-jsl8q" podStartSLOduration=5.621458851 podStartE2EDuration="15.470334719s" totalImagesPullingTime="9.848875868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:09.46972271 +0000 UTC m=+84.039261969" watchObservedRunningTime="2026-09-10 14:49:09.470334719 +0000 UTC m=+84.039873978"
Sep 10 14:49:09.493165 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.493031    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6" podStartSLOduration=2.792664444 podStartE2EDuration="12.492981595s" totalImagesPullingTime="9.700317151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:09.492763441 +0000 UTC m=+84.062302703" watchObservedRunningTime="2026-09-10 14:49:09.492981595 +0000 UTC m=+84.062520855"
Sep 10 14:49:09.518615 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.518554    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6476cbccb4-lhsk6" podStartSLOduration=8.518532224 podStartE2EDuration="8.518532224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:09.518023117 +0000 UTC m=+84.087562401" watchObservedRunningTime="2026-09-10 14:49:09.518532224 +0000 UTC m=+84.088071483"
Sep 10 14:49:09.533894 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:09.533843    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mn2b9" podStartSLOduration=73.685231769 podStartE2EDuration="1m23.533821608s" totalImagesPullingTime="9.848589839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:09.532248189 +0000 UTC m=+84.101787472" watchObservedRunningTime="2026-09-10 14:49:09.533821608 +0000 UTC m=+84.103360868"
Sep 10 14:49:11.423688 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:11.423631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7khtl" event={"ID":"0b6ee187-c103-44b3-80d1-3ce30747905c","Type":"ContainerStarted","Data":"bd378b0c4272f48512ad489a97dc07bafaaffc25c75ef0f07bfb90af95ccee56"}
Sep 10 14:49:11.442266 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:11.442221    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7khtl" podStartSLOduration=23.373708762 podStartE2EDuration="27.44220571s" totalImagesPullingTime="4.068496948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:48:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:11.441338743 +0000 UTC m=+86.010878004" watchObservedRunningTime="2026-09-10 14:49:11.44220571 +0000 UTC m=+86.011744995"
Sep 10 14:49:12.102698 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:12.102666    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:12.102853 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:12.102736    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:12.107471 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:12.107448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:12.431761 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:12.431691    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:49:15.233633 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:15.233598    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:49:18.272162 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:18.272124    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:49:18.272624 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:18.272187    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:49:34.429685 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.429595    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7995c98d7b-jsl8q" podUID="82b83f2b-6a77-49dc-974f-20b02c22c495" containerName="console" containerID="cri-o://d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e" gracePeriod=15
Sep 10 14:49:34.708339 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.708271    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:49:34.775565 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-oauth-serving-cert\") pod \"82b83f2b-6a77-49dc-974f-20b02c22c495\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") "
Sep 10 14:49:34.775758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775579    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-service-ca\") pod \"82b83f2b-6a77-49dc-974f-20b02c22c495\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") "
Sep 10 14:49:34.775758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775616    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-serving-cert\") pod \"82b83f2b-6a77-49dc-974f-20b02c22c495\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") "
Sep 10 14:49:34.775758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775655    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w29l\" (UniqueName: \"kubernetes.io/projected/82b83f2b-6a77-49dc-974f-20b02c22c495-kube-api-access-2w29l\") pod \"82b83f2b-6a77-49dc-974f-20b02c22c495\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") "
Sep 10 14:49:34.775758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775709    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-trusted-ca-bundle\") pod \"82b83f2b-6a77-49dc-974f-20b02c22c495\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") "
Sep 10 14:49:34.775758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775749    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-console-config\") pod \"82b83f2b-6a77-49dc-974f-20b02c22c495\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") "
Sep 10 14:49:34.776017 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-oauth-config\") pod \"82b83f2b-6a77-49dc-974f-20b02c22c495\" (UID: \"82b83f2b-6a77-49dc-974f-20b02c22c495\") "
Sep 10 14:49:34.776017 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.775961    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-service-ca" (OuterVolumeSpecName: "service-ca") pod "82b83f2b-6a77-49dc-974f-20b02c22c495" (UID: "82b83f2b-6a77-49dc-974f-20b02c22c495"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:34.776117 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.776085    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "82b83f2b-6a77-49dc-974f-20b02c22c495" (UID: "82b83f2b-6a77-49dc-974f-20b02c22c495"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:34.776208 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.776184    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-console-config" (OuterVolumeSpecName: "console-config") pod "82b83f2b-6a77-49dc-974f-20b02c22c495" (UID: "82b83f2b-6a77-49dc-974f-20b02c22c495"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:34.776319 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.776288    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "82b83f2b-6a77-49dc-974f-20b02c22c495" (UID: "82b83f2b-6a77-49dc-974f-20b02c22c495"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:49:34.778073 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.778047    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "82b83f2b-6a77-49dc-974f-20b02c22c495" (UID: "82b83f2b-6a77-49dc-974f-20b02c22c495"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:34.778189 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.778145    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b83f2b-6a77-49dc-974f-20b02c22c495-kube-api-access-2w29l" (OuterVolumeSpecName: "kube-api-access-2w29l") pod "82b83f2b-6a77-49dc-974f-20b02c22c495" (UID: "82b83f2b-6a77-49dc-974f-20b02c22c495"). InnerVolumeSpecName "kube-api-access-2w29l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:34.778189 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.778145    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "82b83f2b-6a77-49dc-974f-20b02c22c495" (UID: "82b83f2b-6a77-49dc-974f-20b02c22c495"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:49:34.877247 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.877214    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-trusted-ca-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:49:34.877247 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.877240    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-console-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:49:34.877247 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.877251    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-oauth-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:49:34.877460 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.877259    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-oauth-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:49:34.877460 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.877268    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82b83f2b-6a77-49dc-974f-20b02c22c495-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:49:34.877460 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.877276    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82b83f2b-6a77-49dc-974f-20b02c22c495-console-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:49:34.877460 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:34.877284    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w29l\" (UniqueName: \"kubernetes.io/projected/82b83f2b-6a77-49dc-974f-20b02c22c495-kube-api-access-2w29l\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:49:35.510210 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.510172    2565 generic.go:356] "Generic (PLEG): container finished" podID="82b83f2b-6a77-49dc-974f-20b02c22c495" containerID="d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e" exitCode=0
Sep 10 14:49:35.510576 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.510221    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7995c98d7b-jsl8q"
Sep 10 14:49:35.510576 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.510244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7995c98d7b-jsl8q" event={"ID":"82b83f2b-6a77-49dc-974f-20b02c22c495","Type":"ContainerDied","Data":"d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e"}
Sep 10 14:49:35.510576 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.510294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7995c98d7b-jsl8q" event={"ID":"82b83f2b-6a77-49dc-974f-20b02c22c495","Type":"ContainerDied","Data":"810a379633fe80aef11e5c02dff4c98e13145c8039318901134c5410104eb97e"}
Sep 10 14:49:35.510576 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.510322    2565 scope.go:117] "RemoveContainer" containerID="d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e"
Sep 10 14:49:35.519730 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.519712    2565 scope.go:117] "RemoveContainer" containerID="d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e"
Sep 10 14:49:35.519995 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:49:35.519976    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e\": container with ID starting with d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e not found: ID does not exist" containerID="d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e"
Sep 10 14:49:35.520041 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.520003    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e"} err="failed to get container status \"d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e\": rpc error: code = NotFound desc = could not find container \"d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e\": container with ID starting with d9567b94f6dc3e9ee7ef21347c1772d8706a14c80f4a5f07bb907bf062bc075e not found: ID does not exist"
Sep 10 14:49:35.530012 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.529991    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7995c98d7b-jsl8q"]
Sep 10 14:49:35.535667 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.535635    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7995c98d7b-jsl8q"]
Sep 10 14:49:35.975241 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:35.975212    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b83f2b-6a77-49dc-974f-20b02c22c495" path="/var/lib/kubelet/pods/82b83f2b-6a77-49dc-974f-20b02c22c495/volumes"
Sep 10 14:49:38.276955 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:38.276918    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:49:38.280721 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:38.280701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-559fb774d7-fkhg6"
Sep 10 14:49:40.420701 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:49:40.420668    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mn2b9"
Sep 10 14:50:17.930527 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.930491    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"]
Sep 10 14:50:17.931290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.930827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b83f2b-6a77-49dc-974f-20b02c22c495" containerName="console"
Sep 10 14:50:17.931290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.930844    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b83f2b-6a77-49dc-974f-20b02c22c495" containerName="console"
Sep 10 14:50:17.931290 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.930918    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b83f2b-6a77-49dc-974f-20b02c22c495" containerName="console"
Sep 10 14:50:17.935369 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.935344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:17.937516 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.937487    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 14:50:17.937666 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.937567    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 14:50:17.937760 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.937742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 14:50:17.937930 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.937904    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 14:50:17.938045 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.937994    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 14:50:17.938433 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.938415    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-l28rn\""
Sep 10 14:50:17.948503 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.948476    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 14:50:17.950242 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:17.950128    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"]
Sep 10 14:50:18.016895 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.016858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.017071 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.016900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-serving-certs-ca-bundle\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.017071 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.016935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-secret-telemeter-client\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.017071 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.016958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-telemeter-client-tls\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.017071 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.016972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-federate-client-tls\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.017071 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.016992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.017262 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.017071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-metrics-client-ca\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.017262 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.017110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8ml\" (UniqueName: \"kubernetes.io/projected/a88bb5ae-c13b-493d-b8ad-03158c113d40-kube-api-access-wf8ml\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.117810 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.117766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-secret-telemeter-client\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.117978 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.117820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-telemeter-client-tls\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.118339 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.118305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-federate-client-tls\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.118502 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.118474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.118608 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.118534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-metrics-client-ca\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.118608 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.118578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8ml\" (UniqueName: \"kubernetes.io/projected/a88bb5ae-c13b-493d-b8ad-03158c113d40-kube-api-access-wf8ml\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.118738 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.118700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.118801 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.118741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-serving-certs-ca-bundle\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.119360 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.119334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-metrics-client-ca\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.119578 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.119379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-serving-certs-ca-bundle\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.119716 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.119394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a88bb5ae-c13b-493d-b8ad-03158c113d40-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.120582 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.120554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-telemeter-client-tls\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.120889 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.120866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-secret-telemeter-client\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.121236 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.121213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.121586 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.121568    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a88bb5ae-c13b-493d-b8ad-03158c113d40-federate-client-tls\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.131022 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.130967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8ml\" (UniqueName: \"kubernetes.io/projected/a88bb5ae-c13b-493d-b8ad-03158c113d40-kube-api-access-wf8ml\") pod \"telemeter-client-9749f8bd7-zk2hg\" (UID: \"a88bb5ae-c13b-493d-b8ad-03158c113d40\") " pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.252028 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.251686    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"
Sep 10 14:50:18.532732 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.532655    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9749f8bd7-zk2hg"]
Sep 10 14:50:18.537230 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:50:18.537200    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88bb5ae_c13b_493d_b8ad_03158c113d40.slice/crio-6b5694370a4a8a3501c1aa7bbc66bbb47d408e030321d5a86ba7a81bd5483e48 WatchSource:0}: Error finding container 6b5694370a4a8a3501c1aa7bbc66bbb47d408e030321d5a86ba7a81bd5483e48: Status 404 returned error can't find the container with id 6b5694370a4a8a3501c1aa7bbc66bbb47d408e030321d5a86ba7a81bd5483e48
Sep 10 14:50:18.661491 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:18.661458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg" event={"ID":"a88bb5ae-c13b-493d-b8ad-03158c113d40","Type":"ContainerStarted","Data":"6b5694370a4a8a3501c1aa7bbc66bbb47d408e030321d5a86ba7a81bd5483e48"}
Sep 10 14:50:20.670353 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:20.670317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg" event={"ID":"a88bb5ae-c13b-493d-b8ad-03158c113d40","Type":"ContainerStarted","Data":"b543a18be3bb048b85193fc59274c88b615a32f3c5580ed9a4c6c4b3fa88cdb0"}
Sep 10 14:50:21.676743 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.676667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg" event={"ID":"a88bb5ae-c13b-493d-b8ad-03158c113d40","Type":"ContainerStarted","Data":"d4fcd63d8b37b3ae494bf525b441f265e538759daa4ac33f1399153c374463a7"}
Sep 10 14:50:21.676743 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.676725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg" event={"ID":"a88bb5ae-c13b-493d-b8ad-03158c113d40","Type":"ContainerStarted","Data":"b92ffa197aa894f294b9fe06366e8f22e1a264f8a2576cde0bfed5594434b65b"}
Sep 10 14:50:21.697279 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.697224    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9749f8bd7-zk2hg" podStartSLOduration=1.892581178 podStartE2EDuration="4.697209717s" totalImagesPullingTime="2.804628539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:21.695403978 +0000 UTC m=+156.264943217" watchObservedRunningTime="2026-09-10 14:50:21.697209717 +0000 UTC m=+156.266748976"
Sep 10 14:50:21.778874 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.778850    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f766bc4d7-dh7rw"]
Sep 10 14:50:21.781894 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.781880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.795256 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.792735    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f766bc4d7-dh7rw"]
Sep 10 14:50:21.846412 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.846391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-oauth-serving-cert\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.846523 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.846424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-config\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.846523 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.846452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-serving-cert\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.846696 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.846545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlsgg\" (UniqueName: \"kubernetes.io/projected/838c6d7f-37f0-4600-a802-8d9455c22ed9-kube-api-access-tlsgg\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.846696 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.846592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-oauth-config\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.846696 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.846620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-service-ca\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.846696 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.846667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-trusted-ca-bundle\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.947924 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.947843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-oauth-serving-cert\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.947924 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.947895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-config\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.947930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-serving-cert\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.947959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsgg\" (UniqueName: \"kubernetes.io/projected/838c6d7f-37f0-4600-a802-8d9455c22ed9-kube-api-access-tlsgg\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.947993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-oauth-config\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.948018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-service-ca\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.948063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-trusted-ca-bundle\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948705 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.948682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-oauth-serving-cert\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948812 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.948682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-config\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.948812 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.948766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-service-ca\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.949392 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.949345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-trusted-ca-bundle\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.951118 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.951092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-serving-cert\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.951294 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.951273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-oauth-config\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:21.957732 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:21.957710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlsgg\" (UniqueName: \"kubernetes.io/projected/838c6d7f-37f0-4600-a802-8d9455c22ed9-kube-api-access-tlsgg\") pod \"console-5f766bc4d7-dh7rw\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") " pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:22.091820 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:22.091781    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:22.371066 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:22.371032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f766bc4d7-dh7rw"]
Sep 10 14:50:22.374073 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:50:22.374049    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838c6d7f_37f0_4600_a802_8d9455c22ed9.slice/crio-300af6f11293aa78ecb36baeee05998fd0543c257802baea51961341832383db WatchSource:0}: Error finding container 300af6f11293aa78ecb36baeee05998fd0543c257802baea51961341832383db: Status 404 returned error can't find the container with id 300af6f11293aa78ecb36baeee05998fd0543c257802baea51961341832383db
Sep 10 14:50:22.682860 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:22.682778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f766bc4d7-dh7rw" event={"ID":"838c6d7f-37f0-4600-a802-8d9455c22ed9","Type":"ContainerStarted","Data":"521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a"}
Sep 10 14:50:22.682860 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:22.682839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f766bc4d7-dh7rw" event={"ID":"838c6d7f-37f0-4600-a802-8d9455c22ed9","Type":"ContainerStarted","Data":"300af6f11293aa78ecb36baeee05998fd0543c257802baea51961341832383db"}
Sep 10 14:50:22.699749 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:22.699708    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f766bc4d7-dh7rw" podStartSLOduration=1.6996936599999999 podStartE2EDuration="1.69969366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:22.699006202 +0000 UTC m=+157.268545451" watchObservedRunningTime="2026-09-10 14:50:22.69969366 +0000 UTC m=+157.269232917"
Sep 10 14:50:32.092523 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:32.092484    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:32.093037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:32.092547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:32.097530 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:32.097504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:32.720710 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:32.720680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:50:32.768124 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:32.768082    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6476cbccb4-lhsk6"]
Sep 10 14:50:57.789699 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:57.789577    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6476cbccb4-lhsk6" podUID="b382e9b0-1834-4dd9-8b4d-fc8a5037816b" containerName="console" containerID="cri-o://de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00" gracePeriod=15
Sep 10 14:50:58.026886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.026860    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:50:58.146583 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.146545    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-oauth-serving-cert\") pod \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") "
Sep 10 14:50:58.146583 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.146587    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-service-ca\") pod \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") "
Sep 10 14:50:58.146826 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.146631    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-serving-cert\") pod \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") "
Sep 10 14:50:58.146826 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.146723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-trusted-ca-bundle\") pod \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") "
Sep 10 14:50:58.146826 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.146804    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhxfl\" (UniqueName: \"kubernetes.io/projected/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-kube-api-access-dhxfl\") pod \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") "
Sep 10 14:50:58.146826 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.146828    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-config\") pod \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") "
Sep 10 14:50:58.147072 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.146864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-oauth-config\") pod \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\" (UID: \"b382e9b0-1834-4dd9-8b4d-fc8a5037816b\") "
Sep 10 14:50:58.147072 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.147046    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-service-ca" (OuterVolumeSpecName: "service-ca") pod "b382e9b0-1834-4dd9-8b4d-fc8a5037816b" (UID: "b382e9b0-1834-4dd9-8b4d-fc8a5037816b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:50:58.147072 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.147050    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b382e9b0-1834-4dd9-8b4d-fc8a5037816b" (UID: "b382e9b0-1834-4dd9-8b4d-fc8a5037816b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:50:58.147266 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.147226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b382e9b0-1834-4dd9-8b4d-fc8a5037816b" (UID: "b382e9b0-1834-4dd9-8b4d-fc8a5037816b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:50:58.147348 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.147291    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-config" (OuterVolumeSpecName: "console-config") pod "b382e9b0-1834-4dd9-8b4d-fc8a5037816b" (UID: "b382e9b0-1834-4dd9-8b4d-fc8a5037816b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:50:58.148862 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.148837    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b382e9b0-1834-4dd9-8b4d-fc8a5037816b" (UID: "b382e9b0-1834-4dd9-8b4d-fc8a5037816b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:58.148974 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.148929    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-kube-api-access-dhxfl" (OuterVolumeSpecName: "kube-api-access-dhxfl") pod "b382e9b0-1834-4dd9-8b4d-fc8a5037816b" (UID: "b382e9b0-1834-4dd9-8b4d-fc8a5037816b"). InnerVolumeSpecName "kube-api-access-dhxfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:58.149069 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.149054    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b382e9b0-1834-4dd9-8b4d-fc8a5037816b" (UID: "b382e9b0-1834-4dd9-8b4d-fc8a5037816b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:58.247828 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.247795    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-trusted-ca-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:50:58.247828 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.247822    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhxfl\" (UniqueName: \"kubernetes.io/projected/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-kube-api-access-dhxfl\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:50:58.247828 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.247831    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:50:58.248037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.247840    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-oauth-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:50:58.248037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.247849    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-oauth-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:50:58.248037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.247857    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:50:58.248037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.247865    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b382e9b0-1834-4dd9-8b4d-fc8a5037816b-console-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:50:58.815253 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.815221    2565 generic.go:356] "Generic (PLEG): container finished" podID="b382e9b0-1834-4dd9-8b4d-fc8a5037816b" containerID="de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00" exitCode=0
Sep 10 14:50:58.815664 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.815261    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6476cbccb4-lhsk6"
Sep 10 14:50:58.815664 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.815275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6476cbccb4-lhsk6" event={"ID":"b382e9b0-1834-4dd9-8b4d-fc8a5037816b","Type":"ContainerDied","Data":"de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00"}
Sep 10 14:50:58.815664 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.815315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6476cbccb4-lhsk6" event={"ID":"b382e9b0-1834-4dd9-8b4d-fc8a5037816b","Type":"ContainerDied","Data":"442b9f3c368675460100ded71683b664226d5764c6233f9150260789e14adf36"}
Sep 10 14:50:58.815664 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.815334    2565 scope.go:117] "RemoveContainer" containerID="de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00"
Sep 10 14:50:58.828964 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.828946    2565 scope.go:117] "RemoveContainer" containerID="de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00"
Sep 10 14:50:58.829244 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:50:58.829225    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00\": container with ID starting with de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00 not found: ID does not exist" containerID="de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00"
Sep 10 14:50:58.829305 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.829257    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00"} err="failed to get container status \"de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00\": rpc error: code = NotFound desc = could not find container \"de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00\": container with ID starting with de63adf76720c9ccc8a78a8ca39faf4d0c238ab834f81b9b88120f260decbd00 not found: ID does not exist"
Sep 10 14:50:58.839247 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.839221    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6476cbccb4-lhsk6"]
Sep 10 14:50:58.844622 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:58.844597    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6476cbccb4-lhsk6"]
Sep 10 14:50:59.975937 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:50:59.975904    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b382e9b0-1834-4dd9-8b4d-fc8a5037816b" path="/var/lib/kubelet/pods/b382e9b0-1834-4dd9-8b4d-fc8a5037816b/volumes"
Sep 10 14:51:36.218485 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.218455    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5775cd79-p4qnk"]
Sep 10 14:51:36.218956 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.218802    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b382e9b0-1834-4dd9-8b4d-fc8a5037816b" containerName="console"
Sep 10 14:51:36.218956 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.218816    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b382e9b0-1834-4dd9-8b4d-fc8a5037816b" containerName="console"
Sep 10 14:51:36.218956 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.218873    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b382e9b0-1834-4dd9-8b4d-fc8a5037816b" containerName="console"
Sep 10 14:51:36.222001 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.221981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.232909 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.232888    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5775cd79-p4qnk"]
Sep 10 14:51:36.353475 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.353436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-service-ca\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.353475 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.353476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-trusted-ca-bundle\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.353723 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.353493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-oauth-serving-cert\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.353723 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.353531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs86s\" (UniqueName: \"kubernetes.io/projected/6dd4c689-71bd-4bac-9987-a0bead00ebdc-kube-api-access-vs86s\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.353723 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.353569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-serving-cert\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.353723 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.353615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-oauth-config\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.353723 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.353662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-config\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.454777 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.454741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-serving-cert\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.454970 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.454793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-oauth-config\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.454970 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.454908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-config\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.454970 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.454940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-service-ca\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.454970 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.454959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-trusted-ca-bundle\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.455182 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.454976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-oauth-serving-cert\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.455182 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.455043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs86s\" (UniqueName: \"kubernetes.io/projected/6dd4c689-71bd-4bac-9987-a0bead00ebdc-kube-api-access-vs86s\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.455658 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.455620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-service-ca\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.455778 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.455719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-config\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.456130 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.456105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-oauth-serving-cert\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.456209 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.456196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-trusted-ca-bundle\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.457215 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.457189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-oauth-config\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.457396 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.457378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-serving-cert\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.462184 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.462163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs86s\" (UniqueName: \"kubernetes.io/projected/6dd4c689-71bd-4bac-9987-a0bead00ebdc-kube-api-access-vs86s\") pod \"console-5c5775cd79-p4qnk\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") " pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.531301 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.531217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:36.802839 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.802814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5775cd79-p4qnk"]
Sep 10 14:51:36.805442 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:51:36.805411    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd4c689_71bd_4bac_9987_a0bead00ebdc.slice/crio-3739667af888d2abddfb192382166cdf2c4aad47e2ea8b28053d8144815d449c WatchSource:0}: Error finding container 3739667af888d2abddfb192382166cdf2c4aad47e2ea8b28053d8144815d449c: Status 404 returned error can't find the container with id 3739667af888d2abddfb192382166cdf2c4aad47e2ea8b28053d8144815d449c
Sep 10 14:51:36.951838 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.951800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5775cd79-p4qnk" event={"ID":"6dd4c689-71bd-4bac-9987-a0bead00ebdc","Type":"ContainerStarted","Data":"8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c"}
Sep 10 14:51:36.952004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.951854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5775cd79-p4qnk" event={"ID":"6dd4c689-71bd-4bac-9987-a0bead00ebdc","Type":"ContainerStarted","Data":"3739667af888d2abddfb192382166cdf2c4aad47e2ea8b28053d8144815d449c"}
Sep 10 14:51:36.971238 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:36.971188    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c5775cd79-p4qnk" podStartSLOduration=0.971171726 podStartE2EDuration="971.171726ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:51:36.970107148 +0000 UTC m=+231.539646409" watchObservedRunningTime="2026-09-10 14:51:36.971171726 +0000 UTC m=+231.540711027"
Sep 10 14:51:46.531544 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:46.531501    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:46.531968 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:46.531569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:46.536365 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:46.536342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:46.992846 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:46.992812    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:51:47.070021 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:51:47.069990    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f766bc4d7-dh7rw"]
Sep 10 14:52:12.089273 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.089162    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f766bc4d7-dh7rw" podUID="838c6d7f-37f0-4600-a802-8d9455c22ed9" containerName="console" containerID="cri-o://521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a" gracePeriod=15
Sep 10 14:52:12.319329 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.319304    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:52:12.341659 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.341585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlsgg\" (UniqueName: \"kubernetes.io/projected/838c6d7f-37f0-4600-a802-8d9455c22ed9-kube-api-access-tlsgg\") pod \"838c6d7f-37f0-4600-a802-8d9455c22ed9\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") "
Sep 10 14:52:12.341659 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.341616    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-serving-cert\") pod \"838c6d7f-37f0-4600-a802-8d9455c22ed9\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") "
Sep 10 14:52:12.341659 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.341636    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-oauth-config\") pod \"838c6d7f-37f0-4600-a802-8d9455c22ed9\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") "
Sep 10 14:52:12.342134 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.341680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-service-ca\") pod \"838c6d7f-37f0-4600-a802-8d9455c22ed9\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") "
Sep 10 14:52:12.342134 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.341712    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-oauth-serving-cert\") pod \"838c6d7f-37f0-4600-a802-8d9455c22ed9\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") "
Sep 10 14:52:12.342134 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.341771    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-trusted-ca-bundle\") pod \"838c6d7f-37f0-4600-a802-8d9455c22ed9\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") "
Sep 10 14:52:12.342134 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.341845    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-config\") pod \"838c6d7f-37f0-4600-a802-8d9455c22ed9\" (UID: \"838c6d7f-37f0-4600-a802-8d9455c22ed9\") "
Sep 10 14:52:12.342557 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.342477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-config" (OuterVolumeSpecName: "console-config") pod "838c6d7f-37f0-4600-a802-8d9455c22ed9" (UID: "838c6d7f-37f0-4600-a802-8d9455c22ed9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:52:12.342673 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.342590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "838c6d7f-37f0-4600-a802-8d9455c22ed9" (UID: "838c6d7f-37f0-4600-a802-8d9455c22ed9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:52:12.342753 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.342689    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-service-ca" (OuterVolumeSpecName: "service-ca") pod "838c6d7f-37f0-4600-a802-8d9455c22ed9" (UID: "838c6d7f-37f0-4600-a802-8d9455c22ed9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:52:12.343164 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.343136    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "838c6d7f-37f0-4600-a802-8d9455c22ed9" (UID: "838c6d7f-37f0-4600-a802-8d9455c22ed9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:52:12.344514 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.344491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838c6d7f-37f0-4600-a802-8d9455c22ed9-kube-api-access-tlsgg" (OuterVolumeSpecName: "kube-api-access-tlsgg") pod "838c6d7f-37f0-4600-a802-8d9455c22ed9" (UID: "838c6d7f-37f0-4600-a802-8d9455c22ed9"). InnerVolumeSpecName "kube-api-access-tlsgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:52:12.344746 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.344725    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "838c6d7f-37f0-4600-a802-8d9455c22ed9" (UID: "838c6d7f-37f0-4600-a802-8d9455c22ed9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:52:12.344854 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.344824    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "838c6d7f-37f0-4600-a802-8d9455c22ed9" (UID: "838c6d7f-37f0-4600-a802-8d9455c22ed9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:52:12.442417 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.442386    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:12.442417 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.442411    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlsgg\" (UniqueName: \"kubernetes.io/projected/838c6d7f-37f0-4600-a802-8d9455c22ed9-kube-api-access-tlsgg\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:12.442417 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.442420    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:12.442623 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.442429    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/838c6d7f-37f0-4600-a802-8d9455c22ed9-console-oauth-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:12.442623 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.442437    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:12.442623 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.442445    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-oauth-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:12.442623 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:12.442453    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838c6d7f-37f0-4600-a802-8d9455c22ed9-trusted-ca-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:13.083688 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.083633    2565 generic.go:356] "Generic (PLEG): container finished" podID="838c6d7f-37f0-4600-a802-8d9455c22ed9" containerID="521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a" exitCode=0
Sep 10 14:52:13.083688 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.083687    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f766bc4d7-dh7rw"
Sep 10 14:52:13.083890 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.083687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f766bc4d7-dh7rw" event={"ID":"838c6d7f-37f0-4600-a802-8d9455c22ed9","Type":"ContainerDied","Data":"521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a"}
Sep 10 14:52:13.083890 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.083798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f766bc4d7-dh7rw" event={"ID":"838c6d7f-37f0-4600-a802-8d9455c22ed9","Type":"ContainerDied","Data":"300af6f11293aa78ecb36baeee05998fd0543c257802baea51961341832383db"}
Sep 10 14:52:13.083890 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.083819    2565 scope.go:117] "RemoveContainer" containerID="521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a"
Sep 10 14:52:13.093223 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.093061    2565 scope.go:117] "RemoveContainer" containerID="521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a"
Sep 10 14:52:13.093430 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:52:13.093379    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a\": container with ID starting with 521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a not found: ID does not exist" containerID="521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a"
Sep 10 14:52:13.093430 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.093409    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a"} err="failed to get container status \"521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a\": rpc error: code = NotFound desc = could not find container \"521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a\": container with ID starting with 521ea655a55afdd524fa284136f17d8f740370e2fefcf60ba889bcb4f5251a7a not found: ID does not exist"
Sep 10 14:52:13.105204 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.105178    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f766bc4d7-dh7rw"]
Sep 10 14:52:13.109120 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.109097    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f766bc4d7-dh7rw"]
Sep 10 14:52:13.975371 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:13.975336    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838c6d7f-37f0-4600-a802-8d9455c22ed9" path="/var/lib/kubelet/pods/838c6d7f-37f0-4600-a802-8d9455c22ed9/volumes"
Sep 10 14:52:22.155133 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.155106    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"]
Sep 10 14:52:22.155513 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.155389    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838c6d7f-37f0-4600-a802-8d9455c22ed9" containerName="console"
Sep 10 14:52:22.155513 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.155399    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="838c6d7f-37f0-4600-a802-8d9455c22ed9" containerName="console"
Sep 10 14:52:22.155513 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.155449    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="838c6d7f-37f0-4600-a802-8d9455c22ed9" containerName="console"
Sep 10 14:52:22.159806 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.159786    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.162259 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.162238    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:52:22.163015 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.162995    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lsss\""
Sep 10 14:52:22.163118 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.162995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:52:22.167521 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.167491    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"]
Sep 10 14:52:22.216403 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.216374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6kt\" (UniqueName: \"kubernetes.io/projected/0d41640c-6140-4e62-bba4-175118330f0b-kube-api-access-9v6kt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.216518 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.216421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.216518 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.216455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.317816 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.317789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.317933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.317849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6kt\" (UniqueName: \"kubernetes.io/projected/0d41640c-6140-4e62-bba4-175118330f0b-kube-api-access-9v6kt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.317933 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.317914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.318228 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.318207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.318267 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.318223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.326599 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.326575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6kt\" (UniqueName: \"kubernetes.io/projected/0d41640c-6140-4e62-bba4-175118330f0b-kube-api-access-9v6kt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.470197 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.470147    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:22.738169 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:22.738121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"]
Sep 10 14:52:22.740595 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:52:22.740569    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d41640c_6140_4e62_bba4_175118330f0b.slice/crio-492984d7cc5fc2f8564082cdd5d3195bb5407947dce962dc35414a03c61df91a WatchSource:0}: Error finding container 492984d7cc5fc2f8564082cdd5d3195bb5407947dce962dc35414a03c61df91a: Status 404 returned error can't find the container with id 492984d7cc5fc2f8564082cdd5d3195bb5407947dce962dc35414a03c61df91a
Sep 10 14:52:23.123713 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:23.123675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4" event={"ID":"0d41640c-6140-4e62-bba4-175118330f0b","Type":"ContainerStarted","Data":"492984d7cc5fc2f8564082cdd5d3195bb5407947dce962dc35414a03c61df91a"}
Sep 10 14:52:31.158476 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:31.158442    2565 generic.go:356] "Generic (PLEG): container finished" podID="0d41640c-6140-4e62-bba4-175118330f0b" containerID="b1b2a4eb001ea0fa95c8af6058759d40ec9d0b007f1fd5265656f4f6056c205b" exitCode=0
Sep 10 14:52:31.158476 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:31.158481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4" event={"ID":"0d41640c-6140-4e62-bba4-175118330f0b","Type":"ContainerDied","Data":"b1b2a4eb001ea0fa95c8af6058759d40ec9d0b007f1fd5265656f4f6056c205b"}
Sep 10 14:52:37.188219 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:37.188181    2565 generic.go:356] "Generic (PLEG): container finished" podID="0d41640c-6140-4e62-bba4-175118330f0b" containerID="ee5a7fc0b5f353944c28e8dce7f25a3adddafac901a069ca7079931cbdbc5e9e" exitCode=0
Sep 10 14:52:37.188589 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:37.188269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4" event={"ID":"0d41640c-6140-4e62-bba4-175118330f0b","Type":"ContainerDied","Data":"ee5a7fc0b5f353944c28e8dce7f25a3adddafac901a069ca7079931cbdbc5e9e"}
Sep 10 14:52:44.221338 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:44.221295    2565 generic.go:356] "Generic (PLEG): container finished" podID="0d41640c-6140-4e62-bba4-175118330f0b" containerID="832449c2c6e2af0e7f53b94401a4ec166e6ed1db7cb389523b948d2a4cef5cfe" exitCode=0
Sep 10 14:52:44.221754 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:44.221365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4" event={"ID":"0d41640c-6140-4e62-bba4-175118330f0b","Type":"ContainerDied","Data":"832449c2c6e2af0e7f53b94401a4ec166e6ed1db7cb389523b948d2a4cef5cfe"}
Sep 10 14:52:45.391391 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.391365    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:45.509673 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.509573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-bundle\") pod \"0d41640c-6140-4e62-bba4-175118330f0b\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") "
Sep 10 14:52:45.509673 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.509620    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v6kt\" (UniqueName: \"kubernetes.io/projected/0d41640c-6140-4e62-bba4-175118330f0b-kube-api-access-9v6kt\") pod \"0d41640c-6140-4e62-bba4-175118330f0b\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") "
Sep 10 14:52:45.509673 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.509657    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-util\") pod \"0d41640c-6140-4e62-bba4-175118330f0b\" (UID: \"0d41640c-6140-4e62-bba4-175118330f0b\") "
Sep 10 14:52:45.510247 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.510225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-bundle" (OuterVolumeSpecName: "bundle") pod "0d41640c-6140-4e62-bba4-175118330f0b" (UID: "0d41640c-6140-4e62-bba4-175118330f0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:52:45.511782 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.511759    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d41640c-6140-4e62-bba4-175118330f0b-kube-api-access-9v6kt" (OuterVolumeSpecName: "kube-api-access-9v6kt") pod "0d41640c-6140-4e62-bba4-175118330f0b" (UID: "0d41640c-6140-4e62-bba4-175118330f0b"). InnerVolumeSpecName "kube-api-access-9v6kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:52:45.514555 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.514533    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-util" (OuterVolumeSpecName: "util") pod "0d41640c-6140-4e62-bba4-175118330f0b" (UID: "0d41640c-6140-4e62-bba4-175118330f0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:52:45.611028 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.610991    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:45.611028 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.611021    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v6kt\" (UniqueName: \"kubernetes.io/projected/0d41640c-6140-4e62-bba4-175118330f0b-kube-api-access-9v6kt\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:45.611028 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.611030    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d41640c-6140-4e62-bba4-175118330f0b-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:52:45.861648 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:45.861609    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 14:52:46.231061 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:46.231003    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4"
Sep 10 14:52:46.231209 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:46.231001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406l5j4" event={"ID":"0d41640c-6140-4e62-bba4-175118330f0b","Type":"ContainerDied","Data":"492984d7cc5fc2f8564082cdd5d3195bb5407947dce962dc35414a03c61df91a"}
Sep 10 14:52:46.231209 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:46.231119    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="492984d7cc5fc2f8564082cdd5d3195bb5407947dce962dc35414a03c61df91a"
Sep 10 14:52:50.114617 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114583    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"]
Sep 10 14:52:50.115004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114866    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d41640c-6140-4e62-bba4-175118330f0b" containerName="pull"
Sep 10 14:52:50.115004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114877    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d41640c-6140-4e62-bba4-175118330f0b" containerName="pull"
Sep 10 14:52:50.115004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114896    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d41640c-6140-4e62-bba4-175118330f0b" containerName="util"
Sep 10 14:52:50.115004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114901    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d41640c-6140-4e62-bba4-175118330f0b" containerName="util"
Sep 10 14:52:50.115004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114908    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d41640c-6140-4e62-bba4-175118330f0b" containerName="extract"
Sep 10 14:52:50.115004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114913    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d41640c-6140-4e62-bba4-175118330f0b" containerName="extract"
Sep 10 14:52:50.115004 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.114957    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d41640c-6140-4e62-bba4-175118330f0b" containerName="extract"
Sep 10 14:52:50.116859 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.116843    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.119207 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.119187    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:52:50.119371 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.119355    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rpnnv\""
Sep 10 14:52:50.119423 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.119388    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 14:52:50.131495 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.131475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"]
Sep 10 14:52:50.250452 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.250417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b124a249-93d2-4668-a72d-e7ff8a33724b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jq4h\" (UID: \"b124a249-93d2-4668-a72d-e7ff8a33724b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.250595 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.250489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9p49\" (UniqueName: \"kubernetes.io/projected/b124a249-93d2-4668-a72d-e7ff8a33724b-kube-api-access-q9p49\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jq4h\" (UID: \"b124a249-93d2-4668-a72d-e7ff8a33724b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.351162 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.351124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b124a249-93d2-4668-a72d-e7ff8a33724b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jq4h\" (UID: \"b124a249-93d2-4668-a72d-e7ff8a33724b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.351330 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.351182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9p49\" (UniqueName: \"kubernetes.io/projected/b124a249-93d2-4668-a72d-e7ff8a33724b-kube-api-access-q9p49\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jq4h\" (UID: \"b124a249-93d2-4668-a72d-e7ff8a33724b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.351537 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.351501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b124a249-93d2-4668-a72d-e7ff8a33724b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jq4h\" (UID: \"b124a249-93d2-4668-a72d-e7ff8a33724b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.361966 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.361940    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9p49\" (UniqueName: \"kubernetes.io/projected/b124a249-93d2-4668-a72d-e7ff8a33724b-kube-api-access-q9p49\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jq4h\" (UID: \"b124a249-93d2-4668-a72d-e7ff8a33724b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.425366 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.425285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"
Sep 10 14:52:50.698020 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.697995    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h"]
Sep 10 14:52:50.699902 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:52:50.699874    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb124a249_93d2_4668_a72d_e7ff8a33724b.slice/crio-fbb47e0b274f06e21882996b6ce73de7cc69bac3d545c3809ff15a6500ee1f02 WatchSource:0}: Error finding container fbb47e0b274f06e21882996b6ce73de7cc69bac3d545c3809ff15a6500ee1f02: Status 404 returned error can't find the container with id fbb47e0b274f06e21882996b6ce73de7cc69bac3d545c3809ff15a6500ee1f02
Sep 10 14:52:50.702686 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:50.702668    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:52:51.250218 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:51.250181    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h" event={"ID":"b124a249-93d2-4668-a72d-e7ff8a33724b","Type":"ContainerStarted","Data":"fbb47e0b274f06e21882996b6ce73de7cc69bac3d545c3809ff15a6500ee1f02"}
Sep 10 14:52:53.264220 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:53.263594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h" event={"ID":"b124a249-93d2-4668-a72d-e7ff8a33724b","Type":"ContainerStarted","Data":"95d25122f8d2d04d8bb42cf2c42fec81790a6df0a875e1bc9f159ad8493c8e4b"}
Sep 10 14:52:53.287479 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:53.287414    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jq4h" podStartSLOduration=1.449702805 podStartE2EDuration="3.287397857s" totalImagesPullingTime="1.837695052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:52:53.286501208 +0000 UTC m=+307.856040465" watchObservedRunningTime="2026-09-10 14:52:53.287397857 +0000 UTC m=+307.856937114"
Sep 10 14:52:54.225994 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.225960    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"]
Sep 10 14:52:54.229368 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.229351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.231522 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.231502    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lsss\""
Sep 10 14:52:54.231658 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.231542    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:52:54.231658 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.231589    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:52:54.239102 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.239080    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"]
Sep 10 14:52:54.386845 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.386810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9rhx\" (UniqueName: \"kubernetes.io/projected/110693aa-6bb1-47dd-86d3-2b704f5c28f9-kube-api-access-q9rhx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.387326 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.386857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.387326 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.386886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.488186 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.488083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.488186 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.488130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.488421 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.488237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9rhx\" (UniqueName: \"kubernetes.io/projected/110693aa-6bb1-47dd-86d3-2b704f5c28f9-kube-api-access-q9rhx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.488512 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.488491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.488564 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.488519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.500275 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.500248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9rhx\" (UniqueName: \"kubernetes.io/projected/110693aa-6bb1-47dd-86d3-2b704f5c28f9-kube-api-access-q9rhx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.538878 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.538838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:52:54.809246 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:54.809221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"]
Sep 10 14:52:54.811131 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:52:54.811099    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110693aa_6bb1_47dd_86d3_2b704f5c28f9.slice/crio-28e3bd1ad2a6016ae4680329ea78b047ded1dfb75267ffd8cd751cb468eecc93 WatchSource:0}: Error finding container 28e3bd1ad2a6016ae4680329ea78b047ded1dfb75267ffd8cd751cb468eecc93: Status 404 returned error can't find the container with id 28e3bd1ad2a6016ae4680329ea78b047ded1dfb75267ffd8cd751cb468eecc93
Sep 10 14:52:55.131111 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.131078    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"]
Sep 10 14:52:55.134220 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.134204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.136515 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.136494    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 14:52:55.138205 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.138188    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 14:52:55.138293 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.138188    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cjhmw\""
Sep 10 14:52:55.141364 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.141341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"]
Sep 10 14:52:55.273046 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.273013    2565 generic.go:356] "Generic (PLEG): container finished" podID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerID="0fe4a0af0c6a1772ba1fb008ea7513ad82d4226488432514a2fce1686de39475" exitCode=0
Sep 10 14:52:55.273221 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.273054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5" event={"ID":"110693aa-6bb1-47dd-86d3-2b704f5c28f9","Type":"ContainerDied","Data":"0fe4a0af0c6a1772ba1fb008ea7513ad82d4226488432514a2fce1686de39475"}
Sep 10 14:52:55.273221 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.273101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5" event={"ID":"110693aa-6bb1-47dd-86d3-2b704f5c28f9","Type":"ContainerStarted","Data":"28e3bd1ad2a6016ae4680329ea78b047ded1dfb75267ffd8cd751cb468eecc93"}
Sep 10 14:52:55.294368 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.294343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz4dx\" (UniqueName: \"kubernetes.io/projected/92eae0f5-8528-4c83-888d-2186a17ecf44-kube-api-access-xz4dx\") pod \"cert-manager-cainjector-559cdd8c87-88wlg\" (UID: \"92eae0f5-8528-4c83-888d-2186a17ecf44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.294481 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.294411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92eae0f5-8528-4c83-888d-2186a17ecf44-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-88wlg\" (UID: \"92eae0f5-8528-4c83-888d-2186a17ecf44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.394865 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.394773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz4dx\" (UniqueName: \"kubernetes.io/projected/92eae0f5-8528-4c83-888d-2186a17ecf44-kube-api-access-xz4dx\") pod \"cert-manager-cainjector-559cdd8c87-88wlg\" (UID: \"92eae0f5-8528-4c83-888d-2186a17ecf44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.395223 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.394927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92eae0f5-8528-4c83-888d-2186a17ecf44-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-88wlg\" (UID: \"92eae0f5-8528-4c83-888d-2186a17ecf44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.405447 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.405418    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92eae0f5-8528-4c83-888d-2186a17ecf44-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-88wlg\" (UID: \"92eae0f5-8528-4c83-888d-2186a17ecf44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.405561 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.405511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz4dx\" (UniqueName: \"kubernetes.io/projected/92eae0f5-8528-4c83-888d-2186a17ecf44-kube-api-access-xz4dx\") pod \"cert-manager-cainjector-559cdd8c87-88wlg\" (UID: \"92eae0f5-8528-4c83-888d-2186a17ecf44\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.455355 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.455324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"
Sep 10 14:52:55.722128 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:55.721811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-88wlg"]
Sep 10 14:52:55.724628 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:52:55.724599    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92eae0f5_8528_4c83_888d_2186a17ecf44.slice/crio-e84deec02dc885af97a6cef1d839f5178409b7f6d82015e8506696ca8cc6648b WatchSource:0}: Error finding container e84deec02dc885af97a6cef1d839f5178409b7f6d82015e8506696ca8cc6648b: Status 404 returned error can't find the container with id e84deec02dc885af97a6cef1d839f5178409b7f6d82015e8506696ca8cc6648b
Sep 10 14:52:56.279509 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:56.279470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg" event={"ID":"92eae0f5-8528-4c83-888d-2186a17ecf44","Type":"ContainerStarted","Data":"e84deec02dc885af97a6cef1d839f5178409b7f6d82015e8506696ca8cc6648b"}
Sep 10 14:52:58.291566 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:58.291522    2565 generic.go:356] "Generic (PLEG): container finished" podID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerID="e6dc7eac232dc31ded25cb4b55bfe0631b89c05039e58e92fb1c183a494f4048" exitCode=0
Sep 10 14:52:58.292041 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:58.291573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5" event={"ID":"110693aa-6bb1-47dd-86d3-2b704f5c28f9","Type":"ContainerDied","Data":"e6dc7eac232dc31ded25cb4b55bfe0631b89c05039e58e92fb1c183a494f4048"}
Sep 10 14:52:59.303304 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:59.303271    2565 generic.go:356] "Generic (PLEG): container finished" podID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerID="9addaed358ac5f6f3666d97729cd96d0d24c9b9fdda9412faa612b5841b298b4" exitCode=0
Sep 10 14:52:59.303730 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:59.303350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5" event={"ID":"110693aa-6bb1-47dd-86d3-2b704f5c28f9","Type":"ContainerDied","Data":"9addaed358ac5f6f3666d97729cd96d0d24c9b9fdda9412faa612b5841b298b4"}
Sep 10 14:52:59.306778 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:59.306424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg" event={"ID":"92eae0f5-8528-4c83-888d-2186a17ecf44","Type":"ContainerStarted","Data":"f1fe88268074b355560e2d92235d95740044ba7e137b1377b322824c3b6d4eee"}
Sep 10 14:52:59.340446 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:52:59.340390    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-88wlg" podStartSLOduration=1.577167392 podStartE2EDuration="4.340375764s" totalImagesPullingTime="2.763208372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:52:59.339135393 +0000 UTC m=+313.908674652" watchObservedRunningTime="2026-09-10 14:52:59.340375764 +0000 UTC m=+313.909915021"
Sep 10 14:53:00.432614 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.432591    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:53:00.538758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.538717    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-bundle\") pod \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") "
Sep 10 14:53:00.538953 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.538789    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-util\") pod \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") "
Sep 10 14:53:00.538953 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.538843    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9rhx\" (UniqueName: \"kubernetes.io/projected/110693aa-6bb1-47dd-86d3-2b704f5c28f9-kube-api-access-q9rhx\") pod \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\" (UID: \"110693aa-6bb1-47dd-86d3-2b704f5c28f9\") "
Sep 10 14:53:00.539198 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.539176    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-bundle" (OuterVolumeSpecName: "bundle") pod "110693aa-6bb1-47dd-86d3-2b704f5c28f9" (UID: "110693aa-6bb1-47dd-86d3-2b704f5c28f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:00.540875 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.540852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110693aa-6bb1-47dd-86d3-2b704f5c28f9-kube-api-access-q9rhx" (OuterVolumeSpecName: "kube-api-access-q9rhx") pod "110693aa-6bb1-47dd-86d3-2b704f5c28f9" (UID: "110693aa-6bb1-47dd-86d3-2b704f5c28f9"). InnerVolumeSpecName "kube-api-access-q9rhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:00.544037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.543997    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-util" (OuterVolumeSpecName: "util") pod "110693aa-6bb1-47dd-86d3-2b704f5c28f9" (UID: "110693aa-6bb1-47dd-86d3-2b704f5c28f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:00.640439 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.640399    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:00.640439 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.640430    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9rhx\" (UniqueName: \"kubernetes.io/projected/110693aa-6bb1-47dd-86d3-2b704f5c28f9-kube-api-access-q9rhx\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:00.640439 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:00.640441    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/110693aa-6bb1-47dd-86d3-2b704f5c28f9-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:01.317103 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:01.317071    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5"
Sep 10 14:53:01.317103 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:01.317087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f76sc5" event={"ID":"110693aa-6bb1-47dd-86d3-2b704f5c28f9","Type":"ContainerDied","Data":"28e3bd1ad2a6016ae4680329ea78b047ded1dfb75267ffd8cd751cb468eecc93"}
Sep 10 14:53:01.317320 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:01.317144    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28e3bd1ad2a6016ae4680329ea78b047ded1dfb75267ffd8cd751cb468eecc93"
Sep 10 14:53:06.840338 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840301    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"]
Sep 10 14:53:06.840745 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840598    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerName="util"
Sep 10 14:53:06.840745 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840610    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerName="util"
Sep 10 14:53:06.840745 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840621    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerName="pull"
Sep 10 14:53:06.840745 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840627    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerName="pull"
Sep 10 14:53:06.840745 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840666    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerName="extract"
Sep 10 14:53:06.840745 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840675    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerName="extract"
Sep 10 14:53:06.840745 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.840733    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="110693aa-6bb1-47dd-86d3-2b704f5c28f9" containerName="extract"
Sep 10 14:53:06.844678 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.844660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:06.847243 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.847212    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 14:53:06.847765 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.847747    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:53:06.847851 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.847763    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gptvl\""
Sep 10 14:53:06.850883 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.850853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"]
Sep 10 14:53:06.892064 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.892022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e275381-d069-47ea-9eaa-680f92d588ef-tmp\") pod \"openshift-lws-operator-fd8ccff4c-66dtj\" (UID: \"0e275381-d069-47ea-9eaa-680f92d588ef\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:06.892064 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.892056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lldgz\" (UniqueName: \"kubernetes.io/projected/0e275381-d069-47ea-9eaa-680f92d588ef-kube-api-access-lldgz\") pod \"openshift-lws-operator-fd8ccff4c-66dtj\" (UID: \"0e275381-d069-47ea-9eaa-680f92d588ef\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:06.992701 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.992665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e275381-d069-47ea-9eaa-680f92d588ef-tmp\") pod \"openshift-lws-operator-fd8ccff4c-66dtj\" (UID: \"0e275381-d069-47ea-9eaa-680f92d588ef\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:06.992886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.992787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldgz\" (UniqueName: \"kubernetes.io/projected/0e275381-d069-47ea-9eaa-680f92d588ef-kube-api-access-lldgz\") pod \"openshift-lws-operator-fd8ccff4c-66dtj\" (UID: \"0e275381-d069-47ea-9eaa-680f92d588ef\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:06.993245 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:06.993219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e275381-d069-47ea-9eaa-680f92d588ef-tmp\") pod \"openshift-lws-operator-fd8ccff4c-66dtj\" (UID: \"0e275381-d069-47ea-9eaa-680f92d588ef\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:07.002767 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:07.002742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldgz\" (UniqueName: \"kubernetes.io/projected/0e275381-d069-47ea-9eaa-680f92d588ef-kube-api-access-lldgz\") pod \"openshift-lws-operator-fd8ccff4c-66dtj\" (UID: \"0e275381-d069-47ea-9eaa-680f92d588ef\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:07.154309 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:07.154276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"
Sep 10 14:53:07.425224 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:07.425197    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj"]
Sep 10 14:53:07.427249 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:53:07.427216    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e275381_d069_47ea_9eaa_680f92d588ef.slice/crio-e5b2223c3b2127bcf912fbd15538b427baca582c06104a835bd2d3e400f797d2 WatchSource:0}: Error finding container e5b2223c3b2127bcf912fbd15538b427baca582c06104a835bd2d3e400f797d2: Status 404 returned error can't find the container with id e5b2223c3b2127bcf912fbd15538b427baca582c06104a835bd2d3e400f797d2
Sep 10 14:53:08.346231 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:08.346189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj" event={"ID":"0e275381-d069-47ea-9eaa-680f92d588ef","Type":"ContainerStarted","Data":"e5b2223c3b2127bcf912fbd15538b427baca582c06104a835bd2d3e400f797d2"}
Sep 10 14:53:11.360586 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:11.360551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj" event={"ID":"0e275381-d069-47ea-9eaa-680f92d588ef","Type":"ContainerStarted","Data":"7593545a9e59a2276e08035998747657604c34acd1e48c5968942d6876f38117"}
Sep 10 14:53:11.377404 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:11.377352    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-66dtj" podStartSLOduration=2.506297238 podStartE2EDuration="5.377335796s" totalImagesPullingTime="2.871038558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:11.37598002 +0000 UTC m=+325.945519278" watchObservedRunningTime="2026-09-10 14:53:11.377335796 +0000 UTC m=+325.946875053"
Sep 10 14:53:14.194955 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.194915    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"]
Sep 10 14:53:14.197190 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.197172    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.199387 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.199360    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:53:14.199497 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.199399    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lsss\""
Sep 10 14:53:14.200016 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.200001    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:53:14.206856 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.206834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"]
Sep 10 14:53:14.252324 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.252296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.252442 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.252330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.252442 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.252398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psplf\" (UniqueName: \"kubernetes.io/projected/fcefb55a-7e40-4dcd-95aa-29da3d09df70-kube-api-access-psplf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.353094 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.353061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psplf\" (UniqueName: \"kubernetes.io/projected/fcefb55a-7e40-4dcd-95aa-29da3d09df70-kube-api-access-psplf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.353267 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.353112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.353267 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.353137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.353459 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.353444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.353501 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.353484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.361017 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.360985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psplf\" (UniqueName: \"kubernetes.io/projected/fcefb55a-7e40-4dcd-95aa-29da3d09df70-kube-api-access-psplf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.506121 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.506031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:14.772834 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:14.772803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"]
Sep 10 14:53:14.774880 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:53:14.774850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcefb55a_7e40_4dcd_95aa_29da3d09df70.slice/crio-b24df29de64721ae307218b552d0a5046e408681849b5cbba4233de2e6d2ab20 WatchSource:0}: Error finding container b24df29de64721ae307218b552d0a5046e408681849b5cbba4233de2e6d2ab20: Status 404 returned error can't find the container with id b24df29de64721ae307218b552d0a5046e408681849b5cbba4233de2e6d2ab20
Sep 10 14:53:15.384146 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:15.384110    2565 generic.go:356] "Generic (PLEG): container finished" podID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerID="c4429837c3fcef0fd7704e697381e2902a01eb5887893dd01778ea1cbd1c0df0" exitCode=0
Sep 10 14:53:15.384571 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:15.384199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq" event={"ID":"fcefb55a-7e40-4dcd-95aa-29da3d09df70","Type":"ContainerDied","Data":"c4429837c3fcef0fd7704e697381e2902a01eb5887893dd01778ea1cbd1c0df0"}
Sep 10 14:53:15.384571 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:15.384252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq" event={"ID":"fcefb55a-7e40-4dcd-95aa-29da3d09df70","Type":"ContainerStarted","Data":"b24df29de64721ae307218b552d0a5046e408681849b5cbba4233de2e6d2ab20"}
Sep 10 14:53:16.389921 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:16.389884    2565 generic.go:356] "Generic (PLEG): container finished" podID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerID="03016bba9d8f74fad9523f3937e79aa6f8c734bd7b0af61293ee18215e6b8c6f" exitCode=0
Sep 10 14:53:16.390304 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:16.389974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq" event={"ID":"fcefb55a-7e40-4dcd-95aa-29da3d09df70","Type":"ContainerDied","Data":"03016bba9d8f74fad9523f3937e79aa6f8c734bd7b0af61293ee18215e6b8c6f"}
Sep 10 14:53:17.396694 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:17.396658    2565 generic.go:356] "Generic (PLEG): container finished" podID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerID="cef7bd7f07c1cb888f8c4b21b4b3bc31c5fe79ff247680fd1cee444aec387b85" exitCode=0
Sep 10 14:53:17.397055 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:17.396732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq" event={"ID":"fcefb55a-7e40-4dcd-95aa-29da3d09df70","Type":"ContainerDied","Data":"cef7bd7f07c1cb888f8c4b21b4b3bc31c5fe79ff247680fd1cee444aec387b85"}
Sep 10 14:53:18.532732 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.532708    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:18.586678 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.586627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-util\") pod \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") "
Sep 10 14:53:18.586678 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.586680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-bundle\") pod \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") "
Sep 10 14:53:18.586857 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.586725    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psplf\" (UniqueName: \"kubernetes.io/projected/fcefb55a-7e40-4dcd-95aa-29da3d09df70-kube-api-access-psplf\") pod \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\" (UID: \"fcefb55a-7e40-4dcd-95aa-29da3d09df70\") "
Sep 10 14:53:18.587353 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.587329    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-bundle" (OuterVolumeSpecName: "bundle") pod "fcefb55a-7e40-4dcd-95aa-29da3d09df70" (UID: "fcefb55a-7e40-4dcd-95aa-29da3d09df70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:18.588872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.588842    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcefb55a-7e40-4dcd-95aa-29da3d09df70-kube-api-access-psplf" (OuterVolumeSpecName: "kube-api-access-psplf") pod "fcefb55a-7e40-4dcd-95aa-29da3d09df70" (UID: "fcefb55a-7e40-4dcd-95aa-29da3d09df70"). InnerVolumeSpecName "kube-api-access-psplf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:18.592009 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.591977    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-util" (OuterVolumeSpecName: "util") pod "fcefb55a-7e40-4dcd-95aa-29da3d09df70" (UID: "fcefb55a-7e40-4dcd-95aa-29da3d09df70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:18.688251 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.688164    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:18.688251 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.688192    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcefb55a-7e40-4dcd-95aa-29da3d09df70-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:18.688251 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:18.688205    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psplf\" (UniqueName: \"kubernetes.io/projected/fcefb55a-7e40-4dcd-95aa-29da3d09df70-kube-api-access-psplf\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:19.407517 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:19.407476    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq" event={"ID":"fcefb55a-7e40-4dcd-95aa-29da3d09df70","Type":"ContainerDied","Data":"b24df29de64721ae307218b552d0a5046e408681849b5cbba4233de2e6d2ab20"}
Sep 10 14:53:19.407697 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:19.407525    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrr7jq"
Sep 10 14:53:19.407697 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:19.407537    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b24df29de64721ae307218b552d0a5046e408681849b5cbba4233de2e6d2ab20"
Sep 10 14:53:25.003686 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.003633    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"]
Sep 10 14:53:25.004140 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.004093    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerName="util"
Sep 10 14:53:25.004140 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.004111    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerName="util"
Sep 10 14:53:25.004140 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.004131    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerName="extract"
Sep 10 14:53:25.004140 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.004139    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerName="extract"
Sep 10 14:53:25.004354 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.004149    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerName="pull"
Sep 10 14:53:25.004354 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.004157    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerName="pull"
Sep 10 14:53:25.004354 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.004231    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcefb55a-7e40-4dcd-95aa-29da3d09df70" containerName="extract"
Sep 10 14:53:25.007602 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.007579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.021736 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.021717    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:53:25.021820 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.021717    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:53:25.022463 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.022446    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lsss\""
Sep 10 14:53:25.027963 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.027944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"]
Sep 10 14:53:25.035693 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.035668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.035779 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.035745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.035818 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.035776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfx2m\" (UniqueName: \"kubernetes.io/projected/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-kube-api-access-pfx2m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.136395 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.136371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.136509 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.136411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.136509 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.136432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfx2m\" (UniqueName: \"kubernetes.io/projected/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-kube-api-access-pfx2m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.136798 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.136781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.136836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.136790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.144220 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.144201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfx2m\" (UniqueName: \"kubernetes.io/projected/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-kube-api-access-pfx2m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.316508 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.316439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:25.585378 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:25.585354    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"]
Sep 10 14:53:25.587163 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:53:25.587142    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b1972b_6aac_433c_97b9_cc4c3fbe6977.slice/crio-100e3cf95ccc21c694ad66857fe3fe92d985583df387398b3ac6237c10f0145a WatchSource:0}: Error finding container 100e3cf95ccc21c694ad66857fe3fe92d985583df387398b3ac6237c10f0145a: Status 404 returned error can't find the container with id 100e3cf95ccc21c694ad66857fe3fe92d985583df387398b3ac6237c10f0145a
Sep 10 14:53:26.438166 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.438082    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerID="da62fb79f6fb74f4b59dd438fde279b11e9bdc19ff9c79a97f0c5e528094f174" exitCode=0
Sep 10 14:53:26.438166 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.438138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg" event={"ID":"a2b1972b-6aac-433c-97b9-cc4c3fbe6977","Type":"ContainerDied","Data":"da62fb79f6fb74f4b59dd438fde279b11e9bdc19ff9c79a97f0c5e528094f174"}
Sep 10 14:53:26.438619 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.438172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg" event={"ID":"a2b1972b-6aac-433c-97b9-cc4c3fbe6977","Type":"ContainerStarted","Data":"100e3cf95ccc21c694ad66857fe3fe92d985583df387398b3ac6237c10f0145a"}
Sep 10 14:53:26.668853 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.668822    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"]
Sep 10 14:53:26.672115 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.672100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.676033 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.675911    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 14:53:26.676167 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.676035    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 14:53:26.676765 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.676746    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 14:53:26.676961 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.676946    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 14:53:26.677090 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.677070    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-89wd6\""
Sep 10 14:53:26.685151 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.685130    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"]
Sep 10 14:53:26.749690 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.749584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c70f6abc-e0f1-4d53-81eb-02b5981534cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.749690 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.749634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwwn\" (UniqueName: \"kubernetes.io/projected/c70f6abc-e0f1-4d53-81eb-02b5981534cb-kube-api-access-qrwwn\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.749872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.749712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c70f6abc-e0f1-4d53-81eb-02b5981534cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.749872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.749786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c70f6abc-e0f1-4d53-81eb-02b5981534cb-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.749872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.749818    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c70f6abc-e0f1-4d53-81eb-02b5981534cb-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.850960 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.850924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwwn\" (UniqueName: \"kubernetes.io/projected/c70f6abc-e0f1-4d53-81eb-02b5981534cb-kube-api-access-qrwwn\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.851143 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.850970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c70f6abc-e0f1-4d53-81eb-02b5981534cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.851143 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.851001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c70f6abc-e0f1-4d53-81eb-02b5981534cb-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.851143 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.851021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c70f6abc-e0f1-4d53-81eb-02b5981534cb-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.851143 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.851053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c70f6abc-e0f1-4d53-81eb-02b5981534cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.851491 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.851468    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c70f6abc-e0f1-4d53-81eb-02b5981534cb-manifests\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.853459 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.853432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c70f6abc-e0f1-4d53-81eb-02b5981534cb-tmp\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.853586 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.853563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c70f6abc-e0f1-4d53-81eb-02b5981534cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.853718 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.853611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c70f6abc-e0f1-4d53-81eb-02b5981534cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.863174 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.863153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwwn\" (UniqueName: \"kubernetes.io/projected/c70f6abc-e0f1-4d53-81eb-02b5981534cb-kube-api-access-qrwwn\") pod \"opendatahub-operator-controller-manager-759746987c-4xmjg\" (UID: \"c70f6abc-e0f1-4d53-81eb-02b5981534cb\") " pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:26.981872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:26.981837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:27.292795 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:27.292772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"]
Sep 10 14:53:27.294788 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:53:27.294758    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70f6abc_e0f1_4d53_81eb_02b5981534cb.slice/crio-618c5ccd126913884f5d9cb551bb7886886b21d8e8784d21d55f6b32a76cf334 WatchSource:0}: Error finding container 618c5ccd126913884f5d9cb551bb7886886b21d8e8784d21d55f6b32a76cf334: Status 404 returned error can't find the container with id 618c5ccd126913884f5d9cb551bb7886886b21d8e8784d21d55f6b32a76cf334
Sep 10 14:53:27.444708 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:27.444673    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerID="bfa57c73d8c9b221a084657ec24dd89f2ac59b99cf1eed8e31a54a1b80af3dc0" exitCode=0
Sep 10 14:53:27.445152 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:27.444750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg" event={"ID":"a2b1972b-6aac-433c-97b9-cc4c3fbe6977","Type":"ContainerDied","Data":"bfa57c73d8c9b221a084657ec24dd89f2ac59b99cf1eed8e31a54a1b80af3dc0"}
Sep 10 14:53:27.446217 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:27.446188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg" event={"ID":"c70f6abc-e0f1-4d53-81eb-02b5981534cb","Type":"ContainerStarted","Data":"618c5ccd126913884f5d9cb551bb7886886b21d8e8784d21d55f6b32a76cf334"}
Sep 10 14:53:28.456232 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:28.456194    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerID="e2de205b17298da9b4426c692d2163c872df833461ad9dfd2f2778e8cae25d7f" exitCode=0
Sep 10 14:53:28.456710 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:28.456241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg" event={"ID":"a2b1972b-6aac-433c-97b9-cc4c3fbe6977","Type":"ContainerDied","Data":"e2de205b17298da9b4426c692d2163c872df833461ad9dfd2f2778e8cae25d7f"}
Sep 10 14:53:30.360395 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.360371    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:30.468693 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.468633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg" event={"ID":"c70f6abc-e0f1-4d53-81eb-02b5981534cb","Type":"ContainerStarted","Data":"aeaea8f55ba8ba51a2c300ed67dd9a1df291dea8b3bceeaa09d3dfbfcf52a483"}
Sep 10 14:53:30.470498 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.470474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg" event={"ID":"a2b1972b-6aac-433c-97b9-cc4c3fbe6977","Type":"ContainerDied","Data":"100e3cf95ccc21c694ad66857fe3fe92d985583df387398b3ac6237c10f0145a"}
Sep 10 14:53:30.470622 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.470508    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="100e3cf95ccc21c694ad66857fe3fe92d985583df387398b3ac6237c10f0145a"
Sep 10 14:53:30.470622 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.470527    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cr2dg"
Sep 10 14:53:30.485705 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.485680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-util\") pod \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") "
Sep 10 14:53:30.485825 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.485748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfx2m\" (UniqueName: \"kubernetes.io/projected/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-kube-api-access-pfx2m\") pod \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") "
Sep 10 14:53:30.485825 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.485785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-bundle\") pod \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\" (UID: \"a2b1972b-6aac-433c-97b9-cc4c3fbe6977\") "
Sep 10 14:53:30.486729 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.486691    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-bundle" (OuterVolumeSpecName: "bundle") pod "a2b1972b-6aac-433c-97b9-cc4c3fbe6977" (UID: "a2b1972b-6aac-433c-97b9-cc4c3fbe6977"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:30.488399 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.488372    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-kube-api-access-pfx2m" (OuterVolumeSpecName: "kube-api-access-pfx2m") pod "a2b1972b-6aac-433c-97b9-cc4c3fbe6977" (UID: "a2b1972b-6aac-433c-97b9-cc4c3fbe6977"). InnerVolumeSpecName "kube-api-access-pfx2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:53:30.492204 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.492178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-util" (OuterVolumeSpecName: "util") pod "a2b1972b-6aac-433c-97b9-cc4c3fbe6977" (UID: "a2b1972b-6aac-433c-97b9-cc4c3fbe6977"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:53:30.586979 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.586935    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfx2m\" (UniqueName: \"kubernetes.io/projected/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-kube-api-access-pfx2m\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:30.586979 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.586973    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:30.586979 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:30.586988    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b1972b-6aac-433c-97b9-cc4c3fbe6977-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:53:32.481272 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:32.481236    2565 generic.go:356] "Generic (PLEG): container finished" podID="c70f6abc-e0f1-4d53-81eb-02b5981534cb" containerID="aeaea8f55ba8ba51a2c300ed67dd9a1df291dea8b3bceeaa09d3dfbfcf52a483" exitCode=0
Sep 10 14:53:32.481624 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:32.481294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg" event={"ID":"c70f6abc-e0f1-4d53-81eb-02b5981534cb","Type":"ContainerDied","Data":"aeaea8f55ba8ba51a2c300ed67dd9a1df291dea8b3bceeaa09d3dfbfcf52a483"}
Sep 10 14:53:33.488374 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:33.488339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg" event={"ID":"c70f6abc-e0f1-4d53-81eb-02b5981534cb","Type":"ContainerStarted","Data":"2c6aeb4870977bad2d7aca09846304581082bd3922bd6f06c36a25f6816ea6d9"}
Sep 10 14:53:33.488853 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:33.488571    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:33.511203 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:33.511154    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg" podStartSLOduration=3.036975039 podStartE2EDuration="7.511140863s" totalImagesPullingTime="3.432013962s" totalInitContainerRuntime="1.042151862s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:33.509068646 +0000 UTC m=+348.078607903" watchObservedRunningTime="2026-09-10 14:53:33.511140863 +0000 UTC m=+348.080680121"
Sep 10 14:53:44.496363 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:44.496289    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-759746987c-4xmjg"
Sep 10 14:53:46.598506 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598470    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7497c74497-mn89w"]
Sep 10 14:53:46.598981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598764    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerName="extract"
Sep 10 14:53:46.598981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598775    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerName="extract"
Sep 10 14:53:46.598981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598786    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerName="util"
Sep 10 14:53:46.598981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598792    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerName="util"
Sep 10 14:53:46.598981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598800    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerName="pull"
Sep 10 14:53:46.598981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598807    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerName="pull"
Sep 10 14:53:46.598981 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.598857    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b1972b-6aac-433c-97b9-cc4c3fbe6977" containerName="extract"
Sep 10 14:53:46.603259 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.603243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.610123 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.610097    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 14:53:46.610249 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.610154    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4kbwp\""
Sep 10 14:53:46.610249 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.610171    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 14:53:46.626839 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.626815    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7497c74497-mn89w"]
Sep 10 14:53:46.727473 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.727441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmf8z\" (UniqueName: \"kubernetes.io/projected/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-kube-api-access-gmf8z\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.727836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.727530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-tmp\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.727836 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.727579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-tls-certs\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.827993 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.827954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmf8z\" (UniqueName: \"kubernetes.io/projected/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-kube-api-access-gmf8z\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.828156 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.828000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-tmp\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.828156 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.828038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-tls-certs\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.830142 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.830116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-tmp\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.830443 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.830422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-tls-certs\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.845890 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.845864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmf8z\" (UniqueName: \"kubernetes.io/projected/21e981f8-6f9c-4c01-b0b9-3c366b9deefb-kube-api-access-gmf8z\") pod \"kube-auth-proxy-7497c74497-mn89w\" (UID: \"21e981f8-6f9c-4c01-b0b9-3c366b9deefb\") " pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.859684 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.859613    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"]
Sep 10 14:53:46.862914 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.862901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:46.865862 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.865842    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 14:53:46.865946 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.865842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 14:53:46.867708 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.867691    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xbw9b\""
Sep 10 14:53:46.867819 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.867800    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 14:53:46.876266 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.876244    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"]
Sep 10 14:53:46.913582 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.913549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w"
Sep 10 14:53:46.928870 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.928846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-manager-config\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:46.928967 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.928893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcktj\" (UniqueName: \"kubernetes.io/projected/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-kube-api-access-vcktj\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:46.929016 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.928963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-cert\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:46.929016 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:46.929007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-metrics-cert\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.029580 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.029546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-manager-config\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.029580 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.029583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcktj\" (UniqueName: \"kubernetes.io/projected/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-kube-api-access-vcktj\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.029823 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.029608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-cert\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.029869 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.029841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-metrics-cert\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.030227 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.030207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-manager-config\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.032075 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.032052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-cert\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.032174 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.032089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-metrics-cert\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.044942 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.044923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcktj\" (UniqueName: \"kubernetes.io/projected/5a527c1f-bc0a-460d-9cff-35d6812eb8b8-kube-api-access-vcktj\") pod \"lws-controller-manager-6469457db4-7tphb\" (UID: \"5a527c1f-bc0a-460d-9cff-35d6812eb8b8\") " pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.172019 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.171958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:47.223974 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.223945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7497c74497-mn89w"]
Sep 10 14:53:47.225167 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:53:47.225139    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e981f8_6f9c_4c01_b0b9_3c366b9deefb.slice/crio-42e6523f947357b536a5aae7ca914cd881f5ca122e9a4f0e18f086321c99ef23 WatchSource:0}: Error finding container 42e6523f947357b536a5aae7ca914cd881f5ca122e9a4f0e18f086321c99ef23: Status 404 returned error can't find the container with id 42e6523f947357b536a5aae7ca914cd881f5ca122e9a4f0e18f086321c99ef23
Sep 10 14:53:47.448735 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.448708    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"]
Sep 10 14:53:47.450498 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:53:47.450469    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a527c1f_bc0a_460d_9cff_35d6812eb8b8.slice/crio-6a9202086af40c183473ab77a898c5891527b16921c7ebbb8e68296dc8c12415 WatchSource:0}: Error finding container 6a9202086af40c183473ab77a898c5891527b16921c7ebbb8e68296dc8c12415: Status 404 returned error can't find the container with id 6a9202086af40c183473ab77a898c5891527b16921c7ebbb8e68296dc8c12415
Sep 10 14:53:47.557024 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.556984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb" event={"ID":"5a527c1f-bc0a-460d-9cff-35d6812eb8b8","Type":"ContainerStarted","Data":"6a9202086af40c183473ab77a898c5891527b16921c7ebbb8e68296dc8c12415"}
Sep 10 14:53:47.558259 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:47.558233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w" event={"ID":"21e981f8-6f9c-4c01-b0b9-3c366b9deefb","Type":"ContainerStarted","Data":"42e6523f947357b536a5aae7ca914cd881f5ca122e9a4f0e18f086321c99ef23"}
Sep 10 14:53:50.580512 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:50.580471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb" event={"ID":"5a527c1f-bc0a-460d-9cff-35d6812eb8b8","Type":"ContainerStarted","Data":"baa3e2ef0de764f679e0213b54a5136d2f6bddde7c28fbaeab4138053160d9db"}
Sep 10 14:53:50.580872 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:50.580584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:53:50.595376 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:50.595332    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb" podStartSLOduration=1.594137812 podStartE2EDuration="4.595318051s" totalImagesPullingTime="3.001180239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:50.594590415 +0000 UTC m=+365.164129665" watchObservedRunningTime="2026-09-10 14:53:50.595318051 +0000 UTC m=+365.164857306"
Sep 10 14:53:51.586885 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:51.586845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w" event={"ID":"21e981f8-6f9c-4c01-b0b9-3c366b9deefb","Type":"ContainerStarted","Data":"f48d3c7672a274bb0dc3b8c7bc64fce947a3c3e360b58de1d611024d6cf24526"}
Sep 10 14:53:51.632513 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:53:51.632456    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7497c74497-mn89w" podStartSLOduration=2.347902554 podStartE2EDuration="5.632435624s" totalImagesPullingTime="3.28453307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:53:51.630586993 +0000 UTC m=+366.200126250" watchObservedRunningTime="2026-09-10 14:53:51.632435624 +0000 UTC m=+366.201974901"
Sep 10 14:54:01.589809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:54:01.589772    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6469457db4-7tphb"
Sep 10 14:55:03.754136 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.754094    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"]
Sep 10 14:55:03.757577 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.757559    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.759939 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.759919    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 14:55:03.760553 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.760536    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 14:55:03.760675 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.760659    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4lsss\""
Sep 10 14:55:03.767390 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.767361    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"]
Sep 10 14:55:03.833213 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.833169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.833385 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.833226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.833385 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.833296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqv4\" (UniqueName: \"kubernetes.io/projected/6d8a5596-9243-4de1-ab9a-bd11e5d45209-kube-api-access-8kqv4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.853294 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.853260    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"]
Sep 10 14:55:03.856702 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.856687    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:03.865772 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.865747    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"]
Sep 10 14:55:03.933943 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.933914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:03.934117 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.933957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.934117 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.934030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.934117 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.934066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc69p\" (UniqueName: \"kubernetes.io/projected/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-kube-api-access-jc69p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:03.934117 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.934092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:03.934264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.934138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqv4\" (UniqueName: \"kubernetes.io/projected/6d8a5596-9243-4de1-ab9a-bd11e5d45209-kube-api-access-8kqv4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.934384 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.934364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.934423 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.934385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.945852 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.945820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqv4\" (UniqueName: \"kubernetes.io/projected/6d8a5596-9243-4de1-ab9a-bd11e5d45209-kube-api-access-8kqv4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:03.950799 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.950774    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"]
Sep 10 14:55:03.954454 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.954438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:03.961686 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:03.961663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"]
Sep 10 14:55:04.035660 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.035556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc69p\" (UniqueName: \"kubernetes.io/projected/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-kube-api-access-jc69p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:04.035660 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.035594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:04.035660 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.035633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblrh\" (UniqueName: \"kubernetes.io/projected/c5937a31-eea4-4462-b659-e34c0fa90a0b-kube-api-access-fblrh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.035886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.035729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:04.035886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.035756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.035886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.035785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.036057 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.036035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:04.036098 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.036047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:04.043359 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.043328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc69p\" (UniqueName: \"kubernetes.io/projected/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-kube-api-access-jc69p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:04.055560 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.055529    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"]
Sep 10 14:55:04.059137 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.059113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.065710 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.065683    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"]
Sep 10 14:55:04.069051 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.069030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:04.137202 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8pwx\" (UniqueName: \"kubernetes.io/projected/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-kube-api-access-t8pwx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.137341 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblrh\" (UniqueName: \"kubernetes.io/projected/c5937a31-eea4-4462-b659-e34c0fa90a0b-kube-api-access-fblrh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.137341 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.137419 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.137419 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.137496 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.137692 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.137761 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.137747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.146173 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.146132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblrh\" (UniqueName: \"kubernetes.io/projected/c5937a31-eea4-4462-b659-e34c0fa90a0b-kube-api-access-fblrh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.166035 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.166005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:04.238533 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.238325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.238533 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.238374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.238533 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.238438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8pwx\" (UniqueName: \"kubernetes.io/projected/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-kube-api-access-t8pwx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.238854 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.238803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.238854 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.238835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.246956 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.246932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8pwx\" (UniqueName: \"kubernetes.io/projected/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-kube-api-access-t8pwx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.263860 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.263830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:04.342156 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.342132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"]
Sep 10 14:55:04.343877 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:55:04.343850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8a5596_9243_4de1_ab9a_bd11e5d45209.slice/crio-49a7ca118be608b50a038a16b764163dc9c633aab639082aabc88c4e407bcdad WatchSource:0}: Error finding container 49a7ca118be608b50a038a16b764163dc9c633aab639082aabc88c4e407bcdad: Status 404 returned error can't find the container with id 49a7ca118be608b50a038a16b764163dc9c633aab639082aabc88c4e407bcdad
Sep 10 14:55:04.369580 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.369549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:04.443307 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.442896    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"]
Sep 10 14:55:04.444856 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:55:04.444826    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22ad81f_2ebc_4b27_b09b_1dc48fb14ae1.slice/crio-197c9ca9e0056b03ac862c02ef44ccd92613ea5cf548bf0c119c4439ab1eb314 WatchSource:0}: Error finding container 197c9ca9e0056b03ac862c02ef44ccd92613ea5cf548bf0c119c4439ab1eb314: Status 404 returned error can't find the container with id 197c9ca9e0056b03ac862c02ef44ccd92613ea5cf548bf0c119c4439ab1eb314
Sep 10 14:55:04.532890 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.532852    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"]
Sep 10 14:55:04.534496 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:55:04.534473    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5937a31_eea4_4462_b659_e34c0fa90a0b.slice/crio-ffba819cd753c1b848bc93ee773f9593f3053ef64401d5dec1ab053d0ffeb9d7 WatchSource:0}: Error finding container ffba819cd753c1b848bc93ee773f9593f3053ef64401d5dec1ab053d0ffeb9d7: Status 404 returned error can't find the container with id ffba819cd753c1b848bc93ee773f9593f3053ef64401d5dec1ab053d0ffeb9d7
Sep 10 14:55:04.641758 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.641605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"]
Sep 10 14:55:04.719624 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:55:04.719588    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a8a494_93e9_497b_b5a0_cbf0d1f2104d.slice/crio-c182f2c5410b851390d4c4f6bdeb8877489f1d6b8529b7236ec2ed965bc0af67 WatchSource:0}: Error finding container c182f2c5410b851390d4c4f6bdeb8877489f1d6b8529b7236ec2ed965bc0af67: Status 404 returned error can't find the container with id c182f2c5410b851390d4c4f6bdeb8877489f1d6b8529b7236ec2ed965bc0af67
Sep 10 14:55:04.911967 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.911932    2565 generic.go:356] "Generic (PLEG): container finished" podID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerID="f30cf5b5fb8c68e8c329b47b0a3c63a5e10427f48d6358c8769cda9abf8fbea6" exitCode=0
Sep 10 14:55:04.912378 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.912025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg" event={"ID":"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1","Type":"ContainerDied","Data":"f30cf5b5fb8c68e8c329b47b0a3c63a5e10427f48d6358c8769cda9abf8fbea6"}
Sep 10 14:55:04.912378 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.912080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg" event={"ID":"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1","Type":"ContainerStarted","Data":"197c9ca9e0056b03ac862c02ef44ccd92613ea5cf548bf0c119c4439ab1eb314"}
Sep 10 14:55:04.913684 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.913662    2565 generic.go:356] "Generic (PLEG): container finished" podID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerID="86320e6c6d37533adfaa6ec512a2e1bfdc2d374ca94945837fa45c9830ccaee6" exitCode=0
Sep 10 14:55:04.913768 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.913748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq" event={"ID":"6d8a5596-9243-4de1-ab9a-bd11e5d45209","Type":"ContainerDied","Data":"86320e6c6d37533adfaa6ec512a2e1bfdc2d374ca94945837fa45c9830ccaee6"}
Sep 10 14:55:04.913820 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.913797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq" event={"ID":"6d8a5596-9243-4de1-ab9a-bd11e5d45209","Type":"ContainerStarted","Data":"49a7ca118be608b50a038a16b764163dc9c633aab639082aabc88c4e407bcdad"}
Sep 10 14:55:04.915521 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.915487    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerID="a904f7998c24fa36e6f7bc4787c4e66ad9ffbbac424164c63842f701a7b0a1f5" exitCode=0
Sep 10 14:55:04.915619 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.915570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm" event={"ID":"c5937a31-eea4-4462-b659-e34c0fa90a0b","Type":"ContainerDied","Data":"a904f7998c24fa36e6f7bc4787c4e66ad9ffbbac424164c63842f701a7b0a1f5"}
Sep 10 14:55:04.915619 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.915605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm" event={"ID":"c5937a31-eea4-4462-b659-e34c0fa90a0b","Type":"ContainerStarted","Data":"ffba819cd753c1b848bc93ee773f9593f3053ef64401d5dec1ab053d0ffeb9d7"}
Sep 10 14:55:04.917377 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.917355    2565 generic.go:356] "Generic (PLEG): container finished" podID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerID="85ceeb796879560465d8b162b95fdc63f8e80be75da6b920de00f5b1bc4dfef2" exitCode=0
Sep 10 14:55:04.917452 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.917407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl" event={"ID":"58a8a494-93e9-497b-b5a0-cbf0d1f2104d","Type":"ContainerDied","Data":"85ceeb796879560465d8b162b95fdc63f8e80be75da6b920de00f5b1bc4dfef2"}
Sep 10 14:55:04.917452 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:04.917433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl" event={"ID":"58a8a494-93e9-497b-b5a0-cbf0d1f2104d","Type":"ContainerStarted","Data":"c182f2c5410b851390d4c4f6bdeb8877489f1d6b8529b7236ec2ed965bc0af67"}
Sep 10 14:55:05.925542 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:05.925507    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerID="2207a9222ef76551661c54ff474252d9071349a24246ea067534d1e74844f181" exitCode=0
Sep 10 14:55:05.925991 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:05.925587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm" event={"ID":"c5937a31-eea4-4462-b659-e34c0fa90a0b","Type":"ContainerDied","Data":"2207a9222ef76551661c54ff474252d9071349a24246ea067534d1e74844f181"}
Sep 10 14:55:05.927687 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:05.927662    2565 generic.go:356] "Generic (PLEG): container finished" podID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerID="67300b5c01047f99727b313b02d05c199b665bc44b92fd13d262ac5fbf10cece" exitCode=0
Sep 10 14:55:05.927795 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:05.927754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl" event={"ID":"58a8a494-93e9-497b-b5a0-cbf0d1f2104d","Type":"ContainerDied","Data":"67300b5c01047f99727b313b02d05c199b665bc44b92fd13d262ac5fbf10cece"}
Sep 10 14:55:06.935105 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.935069    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerID="2b764bade3b493fdf10fc27923e2baeb02b65f46a852508522cbfe22fe6e683c" exitCode=0
Sep 10 14:55:06.935723 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.935145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm" event={"ID":"c5937a31-eea4-4462-b659-e34c0fa90a0b","Type":"ContainerDied","Data":"2b764bade3b493fdf10fc27923e2baeb02b65f46a852508522cbfe22fe6e683c"}
Sep 10 14:55:06.937262 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.937240    2565 generic.go:356] "Generic (PLEG): container finished" podID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerID="9526aade72069acc88de3f06a3922eec81627e434e7fd12d957b5b66975289c4" exitCode=0
Sep 10 14:55:06.937344 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.937322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl" event={"ID":"58a8a494-93e9-497b-b5a0-cbf0d1f2104d","Type":"ContainerDied","Data":"9526aade72069acc88de3f06a3922eec81627e434e7fd12d957b5b66975289c4"}
Sep 10 14:55:06.939114 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.939093    2565 generic.go:356] "Generic (PLEG): container finished" podID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerID="e14e003ad47f149a99ab20589c234089e595e5d03386913c5e0568dd726ac82e" exitCode=0
Sep 10 14:55:06.939208 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.939155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg" event={"ID":"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1","Type":"ContainerDied","Data":"e14e003ad47f149a99ab20589c234089e595e5d03386913c5e0568dd726ac82e"}
Sep 10 14:55:06.941091 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.941072    2565 generic.go:356] "Generic (PLEG): container finished" podID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerID="bfa3c6df00794095277c3c5af3fd3e7bfdf93bfb5dd07334300dc6123f10fda0" exitCode=0
Sep 10 14:55:06.941167 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:06.941149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq" event={"ID":"6d8a5596-9243-4de1-ab9a-bd11e5d45209","Type":"ContainerDied","Data":"bfa3c6df00794095277c3c5af3fd3e7bfdf93bfb5dd07334300dc6123f10fda0"}
Sep 10 14:55:07.949706 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:07.949674    2565 generic.go:356] "Generic (PLEG): container finished" podID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerID="8fb4ad17dd5687cd0eb6b8f71ecd78f1931fae4c61c0ad723526b35c1a55ec34" exitCode=0
Sep 10 14:55:07.950131 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:07.949756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg" event={"ID":"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1","Type":"ContainerDied","Data":"8fb4ad17dd5687cd0eb6b8f71ecd78f1931fae4c61c0ad723526b35c1a55ec34"}
Sep 10 14:55:07.951733 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:07.951704    2565 generic.go:356] "Generic (PLEG): container finished" podID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerID="6ae9f3f753d2db448367a91077886832b08cf2e02610513ebf883e896ff6a0dd" exitCode=0
Sep 10 14:55:07.951847 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:07.951747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq" event={"ID":"6d8a5596-9243-4de1-ab9a-bd11e5d45209","Type":"ContainerDied","Data":"6ae9f3f753d2db448367a91077886832b08cf2e02610513ebf883e896ff6a0dd"}
Sep 10 14:55:08.079669 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.079632    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:08.116132 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.116106    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:08.172874 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.172838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-util\") pod \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") "
Sep 10 14:55:08.172874 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.172878    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8pwx\" (UniqueName: \"kubernetes.io/projected/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-kube-api-access-t8pwx\") pod \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") "
Sep 10 14:55:08.173799 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.173392    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-bundle\") pod \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\" (UID: \"58a8a494-93e9-497b-b5a0-cbf0d1f2104d\") "
Sep 10 14:55:08.177744 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.174166    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-bundle" (OuterVolumeSpecName: "bundle") pod "58a8a494-93e9-497b-b5a0-cbf0d1f2104d" (UID: "58a8a494-93e9-497b-b5a0-cbf0d1f2104d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:08.177744 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.174419    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:08.180323 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.180282    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-kube-api-access-t8pwx" (OuterVolumeSpecName: "kube-api-access-t8pwx") pod "58a8a494-93e9-497b-b5a0-cbf0d1f2104d" (UID: "58a8a494-93e9-497b-b5a0-cbf0d1f2104d"). InnerVolumeSpecName "kube-api-access-t8pwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:08.181127 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.181086    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-util" (OuterVolumeSpecName: "util") pod "58a8a494-93e9-497b-b5a0-cbf0d1f2104d" (UID: "58a8a494-93e9-497b-b5a0-cbf0d1f2104d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:08.275024 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.274937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-bundle\") pod \"c5937a31-eea4-4462-b659-e34c0fa90a0b\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") "
Sep 10 14:55:08.275024 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.274981    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fblrh\" (UniqueName: \"kubernetes.io/projected/c5937a31-eea4-4462-b659-e34c0fa90a0b-kube-api-access-fblrh\") pod \"c5937a31-eea4-4462-b659-e34c0fa90a0b\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") "
Sep 10 14:55:08.275024 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.275015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-util\") pod \"c5937a31-eea4-4462-b659-e34c0fa90a0b\" (UID: \"c5937a31-eea4-4462-b659-e34c0fa90a0b\") "
Sep 10 14:55:08.275299 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.275204    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:08.275299 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.275221    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8pwx\" (UniqueName: \"kubernetes.io/projected/58a8a494-93e9-497b-b5a0-cbf0d1f2104d-kube-api-access-t8pwx\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:08.275505 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.275462    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-bundle" (OuterVolumeSpecName: "bundle") pod "c5937a31-eea4-4462-b659-e34c0fa90a0b" (UID: "c5937a31-eea4-4462-b659-e34c0fa90a0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:08.277182 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.277156    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5937a31-eea4-4462-b659-e34c0fa90a0b-kube-api-access-fblrh" (OuterVolumeSpecName: "kube-api-access-fblrh") pod "c5937a31-eea4-4462-b659-e34c0fa90a0b" (UID: "c5937a31-eea4-4462-b659-e34c0fa90a0b"). InnerVolumeSpecName "kube-api-access-fblrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:08.280179 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.280158    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-util" (OuterVolumeSpecName: "util") pod "c5937a31-eea4-4462-b659-e34c0fa90a0b" (UID: "c5937a31-eea4-4462-b659-e34c0fa90a0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:08.376321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.376287    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:08.376321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.376317    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblrh\" (UniqueName: \"kubernetes.io/projected/c5937a31-eea4-4462-b659-e34c0fa90a0b-kube-api-access-fblrh\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:08.376321 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.376327    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c5937a31-eea4-4462-b659-e34c0fa90a0b-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:08.958868 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.958839    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm"
Sep 10 14:55:08.959297 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.958833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cskskm" event={"ID":"c5937a31-eea4-4462-b659-e34c0fa90a0b","Type":"ContainerDied","Data":"ffba819cd753c1b848bc93ee773f9593f3053ef64401d5dec1ab053d0ffeb9d7"}
Sep 10 14:55:08.959297 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.958991    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffba819cd753c1b848bc93ee773f9593f3053ef64401d5dec1ab053d0ffeb9d7"
Sep 10 14:55:08.960999 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.960969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl"
Sep 10 14:55:08.960999 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.960977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nl2xl" event={"ID":"58a8a494-93e9-497b-b5a0-cbf0d1f2104d","Type":"ContainerDied","Data":"c182f2c5410b851390d4c4f6bdeb8877489f1d6b8529b7236ec2ed965bc0af67"}
Sep 10 14:55:08.961197 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:08.961023    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c182f2c5410b851390d4c4f6bdeb8877489f1d6b8529b7236ec2ed965bc0af67"
Sep 10 14:55:09.097241 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.097212    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:09.124467 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.124437    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:09.283797 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.283714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-util\") pod \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") "
Sep 10 14:55:09.283797 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.283753    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-bundle\") pod \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") "
Sep 10 14:55:09.284029 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.283821    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-bundle\") pod \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") "
Sep 10 14:55:09.284029 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.283874    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kqv4\" (UniqueName: \"kubernetes.io/projected/6d8a5596-9243-4de1-ab9a-bd11e5d45209-kube-api-access-8kqv4\") pod \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\" (UID: \"6d8a5596-9243-4de1-ab9a-bd11e5d45209\") "
Sep 10 14:55:09.284029 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.283897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-util\") pod \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") "
Sep 10 14:55:09.284029 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.283919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc69p\" (UniqueName: \"kubernetes.io/projected/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-kube-api-access-jc69p\") pod \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\" (UID: \"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1\") "
Sep 10 14:55:09.284412 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.284384    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-bundle" (OuterVolumeSpecName: "bundle") pod "6d8a5596-9243-4de1-ab9a-bd11e5d45209" (UID: "6d8a5596-9243-4de1-ab9a-bd11e5d45209"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:09.284658 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.284615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-bundle" (OuterVolumeSpecName: "bundle") pod "c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" (UID: "c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:09.286258 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.286235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-kube-api-access-jc69p" (OuterVolumeSpecName: "kube-api-access-jc69p") pod "c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" (UID: "c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1"). InnerVolumeSpecName "kube-api-access-jc69p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:09.286384 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.286364    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8a5596-9243-4de1-ab9a-bd11e5d45209-kube-api-access-8kqv4" (OuterVolumeSpecName: "kube-api-access-8kqv4") pod "6d8a5596-9243-4de1-ab9a-bd11e5d45209" (UID: "6d8a5596-9243-4de1-ab9a-bd11e5d45209"). InnerVolumeSpecName "kube-api-access-8kqv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:09.289517 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.289481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-util" (OuterVolumeSpecName: "util") pod "c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" (UID: "c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:09.290710 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.290689    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-util" (OuterVolumeSpecName: "util") pod "6d8a5596-9243-4de1-ab9a-bd11e5d45209" (UID: "6d8a5596-9243-4de1-ab9a-bd11e5d45209"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:09.384725 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.384690    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:09.384725 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.384718    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kqv4\" (UniqueName: \"kubernetes.io/projected/6d8a5596-9243-4de1-ab9a-bd11e5d45209-kube-api-access-8kqv4\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:09.384725 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.384728    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:09.384725 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.384737    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc69p\" (UniqueName: \"kubernetes.io/projected/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-kube-api-access-jc69p\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:09.385042 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.384745    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d8a5596-9243-4de1-ab9a-bd11e5d45209-util\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:09.385042 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.384754    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:55:09.967330 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.967297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg" event={"ID":"c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1","Type":"ContainerDied","Data":"197c9ca9e0056b03ac862c02ef44ccd92613ea5cf548bf0c119c4439ab1eb314"}
Sep 10 14:55:09.967330 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.967320    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abfmhg"
Sep 10 14:55:09.967801 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.967342    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="197c9ca9e0056b03ac862c02ef44ccd92613ea5cf548bf0c119c4439ab1eb314"
Sep 10 14:55:09.969435 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.969415    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq"
Sep 10 14:55:09.969568 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.969447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tv6jq" event={"ID":"6d8a5596-9243-4de1-ab9a-bd11e5d45209","Type":"ContainerDied","Data":"49a7ca118be608b50a038a16b764163dc9c633aab639082aabc88c4e407bcdad"}
Sep 10 14:55:09.969568 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:55:09.969480    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a7ca118be608b50a038a16b764163dc9c633aab639082aabc88c4e407bcdad"
Sep 10 14:56:39.432746 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.432668    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.432971    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerName="pull"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.432983    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerName="pull"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.432996    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerName="pull"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433003    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerName="pull"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433010    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433015    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433034    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433039    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433044    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433050    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433055    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433062    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433067    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerName="extract"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433074    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433078    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433085    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433089    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433101    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerName="util"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433106    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerName="pull"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433111    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerName="pull"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433116    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerName="pull"
Sep 10 14:56:39.433160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433121    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerName="pull"
Sep 10 14:56:39.433886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433184    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5937a31-eea4-4462-b659-e34c0fa90a0b" containerName="extract"
Sep 10 14:56:39.433886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433192    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22ad81f-2ebc-4b27-b09b-1dc48fb14ae1" containerName="extract"
Sep 10 14:56:39.433886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433199    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8a5596-9243-4de1-ab9a-bd11e5d45209" containerName="extract"
Sep 10 14:56:39.433886 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.433205    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a8a494-93e9-497b-b5a0-cbf0d1f2104d" containerName="extract"
Sep 10 14:56:39.436213 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.436198    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.438339 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.438310    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 14:56:39.438470 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.438324    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cm4wp\""
Sep 10 14:56:39.438470 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.438370    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 14:56:39.438470 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.438374    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 14:56:39.442540 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.442519    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:56:39.465908 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.465888    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:56:39.500433 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.500394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0aa28884-ead1-4b7e-9f36-3f15e866af02-config-file\") pod \"limitador-limitador-84994f7ffc-2fkpg\" (UID: \"0aa28884-ead1-4b7e-9f36-3f15e866af02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.500433 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.500431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfm2d\" (UniqueName: \"kubernetes.io/projected/0aa28884-ead1-4b7e-9f36-3f15e866af02-kube-api-access-nfm2d\") pod \"limitador-limitador-84994f7ffc-2fkpg\" (UID: \"0aa28884-ead1-4b7e-9f36-3f15e866af02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.601557 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.601526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0aa28884-ead1-4b7e-9f36-3f15e866af02-config-file\") pod \"limitador-limitador-84994f7ffc-2fkpg\" (UID: \"0aa28884-ead1-4b7e-9f36-3f15e866af02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.601557 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.601561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfm2d\" (UniqueName: \"kubernetes.io/projected/0aa28884-ead1-4b7e-9f36-3f15e866af02-kube-api-access-nfm2d\") pod \"limitador-limitador-84994f7ffc-2fkpg\" (UID: \"0aa28884-ead1-4b7e-9f36-3f15e866af02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.602140 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.602121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0aa28884-ead1-4b7e-9f36-3f15e866af02-config-file\") pod \"limitador-limitador-84994f7ffc-2fkpg\" (UID: \"0aa28884-ead1-4b7e-9f36-3f15e866af02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.609042 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.609023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfm2d\" (UniqueName: \"kubernetes.io/projected/0aa28884-ead1-4b7e-9f36-3f15e866af02-kube-api-access-nfm2d\") pod \"limitador-limitador-84994f7ffc-2fkpg\" (UID: \"0aa28884-ead1-4b7e-9f36-3f15e866af02\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.747139 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.747051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:39.853424 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.853388    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2tprr"]
Sep 10 14:56:39.856953 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.856932    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2tprr"
Sep 10 14:56:39.859090 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.859072    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qznsd\""
Sep 10 14:56:39.862496 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.862478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2tprr"]
Sep 10 14:56:39.904167 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:39.904146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn85l\" (UniqueName: \"kubernetes.io/projected/77bc8839-b8f8-4ddc-9a0a-639856107ae0-kube-api-access-jn85l\") pod \"authorino-59c974bd48-2tprr\" (UID: \"77bc8839-b8f8-4ddc-9a0a-639856107ae0\") " pod="kuadrant-system/authorino-59c974bd48-2tprr"
Sep 10 14:56:40.005134 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:40.005072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn85l\" (UniqueName: \"kubernetes.io/projected/77bc8839-b8f8-4ddc-9a0a-639856107ae0-kube-api-access-jn85l\") pod \"authorino-59c974bd48-2tprr\" (UID: \"77bc8839-b8f8-4ddc-9a0a-639856107ae0\") " pod="kuadrant-system/authorino-59c974bd48-2tprr"
Sep 10 14:56:40.012465 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:40.012437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn85l\" (UniqueName: \"kubernetes.io/projected/77bc8839-b8f8-4ddc-9a0a-639856107ae0-kube-api-access-jn85l\") pod \"authorino-59c974bd48-2tprr\" (UID: \"77bc8839-b8f8-4ddc-9a0a-639856107ae0\") " pod="kuadrant-system/authorino-59c974bd48-2tprr"
Sep 10 14:56:40.015495 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:40.015464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:56:40.017340 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:56:40.017316    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa28884_ead1_4b7e_9f36_3f15e866af02.slice/crio-855256d600e0094e7515d2177ba486c184752a4291df1a0b804fa98ae54e99ee WatchSource:0}: Error finding container 855256d600e0094e7515d2177ba486c184752a4291df1a0b804fa98ae54e99ee: Status 404 returned error can't find the container with id 855256d600e0094e7515d2177ba486c184752a4291df1a0b804fa98ae54e99ee
Sep 10 14:56:40.166820 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:40.166786    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2tprr"
Sep 10 14:56:40.402669 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:40.402620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg" event={"ID":"0aa28884-ead1-4b7e-9f36-3f15e866af02","Type":"ContainerStarted","Data":"855256d600e0094e7515d2177ba486c184752a4291df1a0b804fa98ae54e99ee"}
Sep 10 14:56:40.433502 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:40.433478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2tprr"]
Sep 10 14:56:40.435393 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:56:40.435369    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77bc8839_b8f8_4ddc_9a0a_639856107ae0.slice/crio-7311b51a76d320002e4ea2e94e58b6ed79b4878aa44fd53f995fac1b1ac5317a WatchSource:0}: Error finding container 7311b51a76d320002e4ea2e94e58b6ed79b4878aa44fd53f995fac1b1ac5317a: Status 404 returned error can't find the container with id 7311b51a76d320002e4ea2e94e58b6ed79b4878aa44fd53f995fac1b1ac5317a
Sep 10 14:56:41.410704 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:41.410662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2tprr" event={"ID":"77bc8839-b8f8-4ddc-9a0a-639856107ae0","Type":"ContainerStarted","Data":"7311b51a76d320002e4ea2e94e58b6ed79b4878aa44fd53f995fac1b1ac5317a"}
Sep 10 14:56:42.420982 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:42.420945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:56:42.438834 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:42.438781    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg" podStartSLOduration=1.109187502 podStartE2EDuration="3.43875962s" totalImagesPullingTime="2.329572118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:42.436690276 +0000 UTC m=+537.006229537" watchObservedRunningTime="2026-09-10 14:56:42.43875962 +0000 UTC m=+537.008298880"
Sep 10 14:56:43.428437 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:43.428397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg" event={"ID":"0aa28884-ead1-4b7e-9f36-3f15e866af02","Type":"ContainerStarted","Data":"14aa190fddee6c84851867649ee3972a9f72fba9e12944533cce0288e5792e57"}
Sep 10 14:56:43.430440 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:43.430414    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2tprr" event={"ID":"77bc8839-b8f8-4ddc-9a0a-639856107ae0","Type":"ContainerStarted","Data":"c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0"}
Sep 10 14:56:43.444975 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:43.444927    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2tprr" podStartSLOduration=1.567599548 podStartE2EDuration="4.444911006s" totalImagesPullingTime="2.877311458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:43.443848601 +0000 UTC m=+538.013387858" watchObservedRunningTime="2026-09-10 14:56:43.444911006 +0000 UTC m=+538.014450302"
Sep 10 14:56:54.436741 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:56:54.436710    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"
Sep 10 14:57:09.916475 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:09.916437    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2tprr"]
Sep 10 14:57:09.916950 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:09.916839    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2tprr" podUID="77bc8839-b8f8-4ddc-9a0a-639856107ae0" containerName="authorino" containerID="cri-o://c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0" gracePeriod=30
Sep 10 14:57:10.161844 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.161819    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2tprr"
Sep 10 14:57:10.259339 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.259248    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn85l\" (UniqueName: \"kubernetes.io/projected/77bc8839-b8f8-4ddc-9a0a-639856107ae0-kube-api-access-jn85l\") pod \"77bc8839-b8f8-4ddc-9a0a-639856107ae0\" (UID: \"77bc8839-b8f8-4ddc-9a0a-639856107ae0\") "
Sep 10 14:57:10.261409 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.261378    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bc8839-b8f8-4ddc-9a0a-639856107ae0-kube-api-access-jn85l" (OuterVolumeSpecName: "kube-api-access-jn85l") pod "77bc8839-b8f8-4ddc-9a0a-639856107ae0" (UID: "77bc8839-b8f8-4ddc-9a0a-639856107ae0"). InnerVolumeSpecName "kube-api-access-jn85l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:57:10.360027 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.359992    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn85l\" (UniqueName: \"kubernetes.io/projected/77bc8839-b8f8-4ddc-9a0a-639856107ae0-kube-api-access-jn85l\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:57:10.566665 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.566565    2565 generic.go:356] "Generic (PLEG): container finished" podID="77bc8839-b8f8-4ddc-9a0a-639856107ae0" containerID="c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0" exitCode=0
Sep 10 14:57:10.566665 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.566611    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2tprr"
Sep 10 14:57:10.566842 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.566663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2tprr" event={"ID":"77bc8839-b8f8-4ddc-9a0a-639856107ae0","Type":"ContainerDied","Data":"c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0"}
Sep 10 14:57:10.566842 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.566707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2tprr" event={"ID":"77bc8839-b8f8-4ddc-9a0a-639856107ae0","Type":"ContainerDied","Data":"7311b51a76d320002e4ea2e94e58b6ed79b4878aa44fd53f995fac1b1ac5317a"}
Sep 10 14:57:10.566842 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.566723    2565 scope.go:117] "RemoveContainer" containerID="c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0"
Sep 10 14:57:10.575200 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.575183    2565 scope.go:117] "RemoveContainer" containerID="c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0"
Sep 10 14:57:10.575466 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:57:10.575449    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0\": container with ID starting with c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0 not found: ID does not exist" containerID="c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0"
Sep 10 14:57:10.575562 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.575478    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0"} err="failed to get container status \"c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0\": rpc error: code = NotFound desc = could not find container \"c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0\": container with ID starting with c3b694ada887a3171cab5d0091654d1778194e703832051db1dbc7e652b7ede0 not found: ID does not exist"
Sep 10 14:57:10.586156 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.586128    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2tprr"]
Sep 10 14:57:10.592125 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:10.592101    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2tprr"]
Sep 10 14:57:11.976799 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:11.976765    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77bc8839-b8f8-4ddc-9a0a-639856107ae0" path="/var/lib/kubelet/pods/77bc8839-b8f8-4ddc-9a0a-639856107ae0/volumes"
Sep 10 14:57:50.437198 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:57:50.437161    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:58:14.980058 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:14.979982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:58:24.938761 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:24.938729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:58:48.310204 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.310169    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"]
Sep 10 14:58:48.310622 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.310468    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bc8839-b8f8-4ddc-9a0a-639856107ae0" containerName="authorino"
Sep 10 14:58:48.310622 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.310478    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bc8839-b8f8-4ddc-9a0a-639856107ae0" containerName="authorino"
Sep 10 14:58:48.310622 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.310533    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bc8839-b8f8-4ddc-9a0a-639856107ae0" containerName="authorino"
Sep 10 14:58:48.313342 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.313321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.319921 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.319837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5vv4r\""
Sep 10 14:58:48.319921 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.319895    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 14:58:48.320170 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.320151    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 14:58:48.320260 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.320242    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 14:58:48.342920 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.342894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"]
Sep 10 14:58:48.384728 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.384696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mz7j\" (UniqueName: \"kubernetes.io/projected/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-kube-api-access-5mz7j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.384859 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.384740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.384859 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.384836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.384937 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.384884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.384937 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.384905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.384937 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.384926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.485911 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.485887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486023 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.485926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486023 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.485941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486105 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.486057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486105 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.486100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mz7j\" (UniqueName: \"kubernetes.io/projected/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-kube-api-access-5mz7j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486185 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.486126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486329 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.486305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486418 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.486345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.486418 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.486397    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.488364 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.488342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.488500 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.488484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.496201 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.496184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mz7j\" (UniqueName: \"kubernetes.io/projected/ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d-kube-api-access-5mz7j\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9\" (UID: \"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.623668 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.623627    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:48.907626 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.907549    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"]
Sep 10 14:58:48.910371 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:58:48.910345    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba799cc0_e5c6_4fc9_90d5_dafde7a20a6d.slice/crio-3eb1ac56509dfa6b06eafe36b65439bf488df9fe713456ea0b0d12f4dfcde9cf WatchSource:0}: Error finding container 3eb1ac56509dfa6b06eafe36b65439bf488df9fe713456ea0b0d12f4dfcde9cf: Status 404 returned error can't find the container with id 3eb1ac56509dfa6b06eafe36b65439bf488df9fe713456ea0b0d12f4dfcde9cf
Sep 10 14:58:48.911669 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:48.911652    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:58:49.059958 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:49.059924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9" event={"ID":"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d","Type":"ContainerStarted","Data":"3eb1ac56509dfa6b06eafe36b65439bf488df9fe713456ea0b0d12f4dfcde9cf"}
Sep 10 14:58:49.197748 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:49.197634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:58:50.066549 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:50.066508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9" event={"ID":"ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d","Type":"ContainerStarted","Data":"55a83a46df576acd23f56174425e5815a0e342a2ec33999b0397644bff198a9a"}
Sep 10 14:58:50.111760 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:50.111710    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9" podStartSLOduration=1.089011554 podStartE2EDuration="2.111693426s" totalImagesPullingTime="1.022681872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:50.110511517 +0000 UTC m=+664.680050776" watchObservedRunningTime="2026-09-10 14:58:50.111693426 +0000 UTC m=+664.681232684"
Sep 10 14:58:55.312767 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:55.312732    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:58:58.624688 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:58.624653    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:58.625195 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:58.624997    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:58.637246 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:58.637221    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:58:59.125156 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:58:59.125129    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9"
Sep 10 14:59:02.689286 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.689256    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dcd86c77-l8ftj"]
Sep 10 14:59:02.695811 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.695788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.715477 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.715453    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dcd86c77-l8ftj"]
Sep 10 14:59:02.808441 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.808403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-config\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.808441 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.808441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-serving-cert\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.808691 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.808465    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-oauth-serving-cert\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.808691 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.808561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-service-ca\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.808691 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.808596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-trusted-ca-bundle\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.808691 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.808614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k675c\" (UniqueName: \"kubernetes.io/projected/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-kube-api-access-k675c\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.808823 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.808715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-oauth-config\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.910122 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.910083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-serving-cert\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.910122 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.910129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-oauth-serving-cert\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.910388 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.910166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-service-ca\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.910388 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.910185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-trusted-ca-bundle\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.910388 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.910200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k675c\" (UniqueName: \"kubernetes.io/projected/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-kube-api-access-k675c\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.910388 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.910228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-oauth-config\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.910388 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.910275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-config\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.911030 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.911004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-service-ca\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.911148 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.911115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-config\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.911211 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.911173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-trusted-ca-bundle\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.911211 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.911176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-oauth-serving-cert\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.912733 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.912714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-serving-cert\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.912860 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.912839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-console-oauth-config\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:02.924459 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:02.924437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k675c\" (UniqueName: \"kubernetes.io/projected/e36309a2-f08c-4f4f-9ca3-7dca8daf359a-kube-api-access-k675c\") pod \"console-65dcd86c77-l8ftj\" (UID: \"e36309a2-f08c-4f4f-9ca3-7dca8daf359a\") " pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:03.004544 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:03.004454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:03.292561 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:03.292534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dcd86c77-l8ftj"]
Sep 10 14:59:03.293573 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:59:03.293545    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36309a2_f08c_4f4f_9ca3_7dca8daf359a.slice/crio-e8282e660cd11068dc9fd661c8906e14b9a2f9830cf71ef92f77fee5dbe50fb4 WatchSource:0}: Error finding container e8282e660cd11068dc9fd661c8906e14b9a2f9830cf71ef92f77fee5dbe50fb4: Status 404 returned error can't find the container with id e8282e660cd11068dc9fd661c8906e14b9a2f9830cf71ef92f77fee5dbe50fb4
Sep 10 14:59:04.139902 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:04.139859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcd86c77-l8ftj" event={"ID":"e36309a2-f08c-4f4f-9ca3-7dca8daf359a","Type":"ContainerStarted","Data":"c0281a76f050adf8a3b2ec8be43b6c3ac4ebb16999ab9cc39143b19a431bdacf"}
Sep 10 14:59:04.140293 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:04.139922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcd86c77-l8ftj" event={"ID":"e36309a2-f08c-4f4f-9ca3-7dca8daf359a","Type":"ContainerStarted","Data":"e8282e660cd11068dc9fd661c8906e14b9a2f9830cf71ef92f77fee5dbe50fb4"}
Sep 10 14:59:04.212899 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:04.212845    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65dcd86c77-l8ftj" podStartSLOduration=2.21283026 podStartE2EDuration="2.21283026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:04.211921877 +0000 UTC m=+678.781461136" watchObservedRunningTime="2026-09-10 14:59:04.21283026 +0000 UTC m=+678.782369587"
Sep 10 14:59:13.005037 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:13.004993    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:13.005533 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:13.005059    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:13.009934 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:13.009909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:13.190222 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:13.190193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65dcd86c77-l8ftj"
Sep 10 14:59:13.295753 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:13.295671    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5775cd79-p4qnk"]
Sep 10 14:59:23.637813 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.637777    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"]
Sep 10 14:59:23.641203 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.641186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.645345 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.645318    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 14:59:23.670725 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.670704    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"]
Sep 10 14:59:23.696746 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.696725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.696838 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.696755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8s9\" (UniqueName: \"kubernetes.io/projected/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-kube-api-access-9r8s9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.696838 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.696778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.696838 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.696809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.696943 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.696860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.696943 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.696907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798213 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798330 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798330 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8s9\" (UniqueName: \"kubernetes.io/projected/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-kube-api-access-9r8s9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798420 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798420 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798559 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798809 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798928 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.798928 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.798878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.800530 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.800511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.800935 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.800916    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.809442 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.809422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8s9\" (UniqueName: \"kubernetes.io/projected/1c6446fd-b2fb-4363-bc34-3a663fa20a6e-kube-api-access-9r8s9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg\" (UID: \"1c6446fd-b2fb-4363-bc34-3a663fa20a6e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:23.951264 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:23.951192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:24.246082 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:24.246057    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"]
Sep 10 14:59:24.248198 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:59:24.248169    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6446fd_b2fb_4363_bc34_3a663fa20a6e.slice/crio-0c05b5538bfd6b8799b7f39f8049dfd27c3dd5bd11ce5a6307d46b1dd42b766e WatchSource:0}: Error finding container 0c05b5538bfd6b8799b7f39f8049dfd27c3dd5bd11ce5a6307d46b1dd42b766e: Status 404 returned error can't find the container with id 0c05b5538bfd6b8799b7f39f8049dfd27c3dd5bd11ce5a6307d46b1dd42b766e
Sep 10 14:59:25.244579 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:25.244541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg" event={"ID":"1c6446fd-b2fb-4363-bc34-3a663fa20a6e","Type":"ContainerStarted","Data":"3f48418f48bf277c25c74508d0ba5fbfc61feceba8d66f67563d306d20bf0a33"}
Sep 10 14:59:25.245059 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:25.244604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg" event={"ID":"1c6446fd-b2fb-4363-bc34-3a663fa20a6e","Type":"ContainerStarted","Data":"0c05b5538bfd6b8799b7f39f8049dfd27c3dd5bd11ce5a6307d46b1dd42b766e"}
Sep 10 14:59:25.284245 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:25.284197    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg" podStartSLOduration=2.077380099 podStartE2EDuration="2.284181893s" totalImagesPullingTime="206.801794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:25.282555133 +0000 UTC m=+699.852094392" watchObservedRunningTime="2026-09-10 14:59:25.284181893 +0000 UTC m=+699.853721223"
Sep 10 14:59:25.394403 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:25.394367    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:59:30.202060 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.202024    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"]
Sep 10 14:59:30.205530 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.205512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.208887 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.208866    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 14:59:30.223517 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.223486    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"]
Sep 10 14:59:30.253325 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.253292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.253325 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.253331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.253572 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.253354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbnf8\" (UniqueName: \"kubernetes.io/projected/8bd9fd97-3922-4c43-8234-693c485af606-kube-api-access-qbnf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.253572 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.253488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.253572 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.253524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.253572 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.253552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bd9fd97-3922-4c43-8234-693c485af606-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.354916 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.354883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.354924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.354956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnf8\" (UniqueName: \"kubernetes.io/projected/8bd9fd97-3922-4c43-8234-693c485af606-kube-api-access-qbnf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.355014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.355040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355096 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.355064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bd9fd97-3922-4c43-8234-693c485af606-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355377 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.355360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355490 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.355458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.355490 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.355473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.357160 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.357140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bd9fd97-3922-4c43-8234-693c485af606-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.357578 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.357555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bd9fd97-3922-4c43-8234-693c485af606-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.364859 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.364835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnf8\" (UniqueName: \"kubernetes.io/projected/8bd9fd97-3922-4c43-8234-693c485af606-kube-api-access-qbnf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hgrps\" (UID: \"8bd9fd97-3922-4c43-8234-693c485af606\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.516398 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.516314    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:30.795801 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:30.795772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"]
Sep 10 14:59:30.795921 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:59:30.795848    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd9fd97_3922_4c43_8234_693c485af606.slice/crio-db34fd418c113cd11e1651deadbeccdca9f96b3202f6df24f46da039e8bf290a WatchSource:0}: Error finding container db34fd418c113cd11e1651deadbeccdca9f96b3202f6df24f46da039e8bf290a: Status 404 returned error can't find the container with id db34fd418c113cd11e1651deadbeccdca9f96b3202f6df24f46da039e8bf290a
Sep 10 14:59:31.277792 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:31.277756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps" event={"ID":"8bd9fd97-3922-4c43-8234-693c485af606","Type":"ContainerStarted","Data":"50e7e43519b1ce33e89220877478cb5937c260181a2400fe97f470f3d4b3eff3"}
Sep 10 14:59:31.278238 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:31.277811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps" event={"ID":"8bd9fd97-3922-4c43-8234-693c485af606","Type":"ContainerStarted","Data":"db34fd418c113cd11e1651deadbeccdca9f96b3202f6df24f46da039e8bf290a"}
Sep 10 14:59:31.322093 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:31.322040    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps" podStartSLOduration=1.143301801 podStartE2EDuration="1.322026837s" totalImagesPullingTime="178.725036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:31.3212713 +0000 UTC m=+705.890810559" watchObservedRunningTime="2026-09-10 14:59:31.322026837 +0000 UTC m=+705.891566091"
Sep 10 14:59:32.639815 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:32.639779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:59:33.951547 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:33.951511    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:33.951982 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:33.951574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:33.963561 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:33.963536    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:34.304135 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:34.304048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg"
Sep 10 14:59:35.796183 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.796153    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"]
Sep 10 14:59:35.799781 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.799759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:35.802713 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.802673    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 14:59:35.823295 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.823266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"]
Sep 10 14:59:35.903950 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.903922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:35.903950 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.903952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:35.904147 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.903977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e088bb4-4a11-4a55-86be-d790c45cf930-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:35.904147 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.904001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:35.904147 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.904112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29794\" (UniqueName: \"kubernetes.io/projected/9e088bb4-4a11-4a55-86be-d790c45cf930-kube-api-access-29794\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:35.904289 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:35.904151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.005531 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.005505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29794\" (UniqueName: \"kubernetes.io/projected/9e088bb4-4a11-4a55-86be-d790c45cf930-kube-api-access-29794\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.005531 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.005533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.005717 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.005572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.005717 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.005588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.005717 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.005612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e088bb4-4a11-4a55-86be-d790c45cf930-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.005717 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.005635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.006005 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.005985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.006061 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.006029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.006097 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.006068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.007794 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.007778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e088bb4-4a11-4a55-86be-d790c45cf930-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.008063 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.008045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e088bb4-4a11-4a55-86be-d790c45cf930-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.025272 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.025253    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29794\" (UniqueName: \"kubernetes.io/projected/9e088bb4-4a11-4a55-86be-d790c45cf930-kube-api-access-29794\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l\" (UID: \"9e088bb4-4a11-4a55-86be-d790c45cf930\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.109485 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.109459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:36.388984 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:36.388956    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"]
Sep 10 14:59:36.390619 ip-10-0-130-2 kubenswrapper[2565]: W0910 14:59:36.390595    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e088bb4_4a11_4a55_86be_d790c45cf930.slice/crio-bd07e26d37687494d13e97fdbbc01538a5c0af8067468338ba470d0f02c2e4a0 WatchSource:0}: Error finding container bd07e26d37687494d13e97fdbbc01538a5c0af8067468338ba470d0f02c2e4a0: Status 404 returned error can't find the container with id bd07e26d37687494d13e97fdbbc01538a5c0af8067468338ba470d0f02c2e4a0
Sep 10 14:59:37.310492 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:37.310449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l" event={"ID":"9e088bb4-4a11-4a55-86be-d790c45cf930","Type":"ContainerStarted","Data":"6e2ef0d2b4705ab7c24ef6577b071af5348fccc5131505865ccced3f77471da3"}
Sep 10 14:59:37.310871 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:37.310502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l" event={"ID":"9e088bb4-4a11-4a55-86be-d790c45cf930","Type":"ContainerStarted","Data":"bd07e26d37687494d13e97fdbbc01538a5c0af8067468338ba470d0f02c2e4a0"}
Sep 10 14:59:37.352339 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:37.352277    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l" podStartSLOduration=2.157848379 podStartE2EDuration="2.352262512s" totalImagesPullingTime="194.414133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:37.35215103 +0000 UTC m=+711.921690288" watchObservedRunningTime="2026-09-10 14:59:37.352262512 +0000 UTC m=+711.921801769"
Sep 10 14:59:38.316208 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.316167    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c5775cd79-p4qnk" podUID="6dd4c689-71bd-4bac-9987-a0bead00ebdc" containerName="console" containerID="cri-o://8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c" gracePeriod=15
Sep 10 14:59:38.565123 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.565098    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:59:38.622816 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.622784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 14:59:38.627865 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.627837    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs86s\" (UniqueName: \"kubernetes.io/projected/6dd4c689-71bd-4bac-9987-a0bead00ebdc-kube-api-access-vs86s\") pod \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") "
Sep 10 14:59:38.628007 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.627869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-serving-cert\") pod \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") "
Sep 10 14:59:38.628007 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.627888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-oauth-serving-cert\") pod \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") "
Sep 10 14:59:38.628007 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.627922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-service-ca\") pod \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") "
Sep 10 14:59:38.628007 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.627952    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-trusted-ca-bundle\") pod \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") "
Sep 10 14:59:38.628007 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.627995    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-config\") pod \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") "
Sep 10 14:59:38.628381 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.628048    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-oauth-config\") pod \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\" (UID: \"6dd4c689-71bd-4bac-9987-a0bead00ebdc\") "
Sep 10 14:59:38.628458 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.628433    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-service-ca" (OuterVolumeSpecName: "service-ca") pod "6dd4c689-71bd-4bac-9987-a0bead00ebdc" (UID: "6dd4c689-71bd-4bac-9987-a0bead00ebdc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:59:38.628592 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.628575    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6dd4c689-71bd-4bac-9987-a0bead00ebdc" (UID: "6dd4c689-71bd-4bac-9987-a0bead00ebdc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:59:38.629164 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.628940    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-config" (OuterVolumeSpecName: "console-config") pod "6dd4c689-71bd-4bac-9987-a0bead00ebdc" (UID: "6dd4c689-71bd-4bac-9987-a0bead00ebdc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:59:38.629164 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.629136    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6dd4c689-71bd-4bac-9987-a0bead00ebdc" (UID: "6dd4c689-71bd-4bac-9987-a0bead00ebdc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:59:38.630319 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.630281    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd4c689-71bd-4bac-9987-a0bead00ebdc-kube-api-access-vs86s" (OuterVolumeSpecName: "kube-api-access-vs86s") pod "6dd4c689-71bd-4bac-9987-a0bead00ebdc" (UID: "6dd4c689-71bd-4bac-9987-a0bead00ebdc"). InnerVolumeSpecName "kube-api-access-vs86s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:59:38.630424 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.630338    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6dd4c689-71bd-4bac-9987-a0bead00ebdc" (UID: "6dd4c689-71bd-4bac-9987-a0bead00ebdc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:59:38.630683 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.630661    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6dd4c689-71bd-4bac-9987-a0bead00ebdc" (UID: "6dd4c689-71bd-4bac-9987-a0bead00ebdc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:59:38.729056 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.729016    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs86s\" (UniqueName: \"kubernetes.io/projected/6dd4c689-71bd-4bac-9987-a0bead00ebdc-kube-api-access-vs86s\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:59:38.729056 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.729046    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:59:38.729056 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.729054    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-oauth-serving-cert\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:59:38.729056 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.729063    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:59:38.729355 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.729071    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-trusted-ca-bundle\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:59:38.729355 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.729080    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:59:38.729355 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:38.729088    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dd4c689-71bd-4bac-9987-a0bead00ebdc-console-oauth-config\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 14:59:39.322153 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.322122    2565 generic.go:356] "Generic (PLEG): container finished" podID="6dd4c689-71bd-4bac-9987-a0bead00ebdc" containerID="8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c" exitCode=0
Sep 10 14:59:39.322553 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.322174    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5775cd79-p4qnk"
Sep 10 14:59:39.322553 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.322203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5775cd79-p4qnk" event={"ID":"6dd4c689-71bd-4bac-9987-a0bead00ebdc","Type":"ContainerDied","Data":"8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c"}
Sep 10 14:59:39.322553 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.322269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5775cd79-p4qnk" event={"ID":"6dd4c689-71bd-4bac-9987-a0bead00ebdc","Type":"ContainerDied","Data":"3739667af888d2abddfb192382166cdf2c4aad47e2ea8b28053d8144815d449c"}
Sep 10 14:59:39.322553 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.322288    2565 scope.go:117] "RemoveContainer" containerID="8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c"
Sep 10 14:59:39.331122 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.331108    2565 scope.go:117] "RemoveContainer" containerID="8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c"
Sep 10 14:59:39.331391 ip-10-0-130-2 kubenswrapper[2565]: E0910 14:59:39.331365    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c\": container with ID starting with 8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c not found: ID does not exist" containerID="8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c"
Sep 10 14:59:39.331435 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.331403    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c"} err="failed to get container status \"8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c\": rpc error: code = NotFound desc = could not find container \"8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c\": container with ID starting with 8f9b600fa354c2468bd22d141ddba3f6908021641fcedd4cafd4316d97d93c4c not found: ID does not exist"
Sep 10 14:59:39.357313 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.357288    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5775cd79-p4qnk"]
Sep 10 14:59:39.371299 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.371277    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c5775cd79-p4qnk"]
Sep 10 14:59:39.976736 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:39.976704    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd4c689-71bd-4bac-9987-a0bead00ebdc" path="/var/lib/kubelet/pods/6dd4c689-71bd-4bac-9987-a0bead00ebdc/volumes"
Sep 10 14:59:40.517421 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:40.517386    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:40.517832 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:40.517453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:40.530068 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:40.530045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:41.344457 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:41.344430    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hgrps"
Sep 10 14:59:46.110056 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:46.110020    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:46.110454 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:46.110083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:46.122819 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:46.122793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 14:59:46.369738 ip-10-0-130-2 kubenswrapper[2565]: I0910 14:59:46.369711    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l"
Sep 10 15:01:29.194156 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:29.194117    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:01:31.977299 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:31.977271    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7555-eb6312"]
Sep 10 15:01:31.977670 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:31.977578    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd4c689-71bd-4bac-9987-a0bead00ebdc" containerName="console"
Sep 10 15:01:31.977670 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:31.977590    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd4c689-71bd-4bac-9987-a0bead00ebdc" containerName="console"
Sep 10 15:01:31.977670 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:31.977659    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd4c689-71bd-4bac-9987-a0bead00ebdc" containerName="console"
Sep 10 15:01:31.980730 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:31.980715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:31.983667 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:31.983624    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrlvr\""
Sep 10 15:01:31.984629 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:31.984602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7555-eb6312"]
Sep 10 15:01:32.130607 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:32.130561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq96d\" (UniqueName: \"kubernetes.io/projected/e887e08b-60a3-43fe-bd5c-2bd8a843f9b0-kube-api-access-xq96d\") pod \"test-curl-7555-eb6312\" (UID: \"e887e08b-60a3-43fe-bd5c-2bd8a843f9b0\") " pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:32.231890 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:32.231803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq96d\" (UniqueName: \"kubernetes.io/projected/e887e08b-60a3-43fe-bd5c-2bd8a843f9b0-kube-api-access-xq96d\") pod \"test-curl-7555-eb6312\" (UID: \"e887e08b-60a3-43fe-bd5c-2bd8a843f9b0\") " pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:32.242578 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:32.242554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq96d\" (UniqueName: \"kubernetes.io/projected/e887e08b-60a3-43fe-bd5c-2bd8a843f9b0-kube-api-access-xq96d\") pod \"test-curl-7555-eb6312\" (UID: \"e887e08b-60a3-43fe-bd5c-2bd8a843f9b0\") " pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:32.290760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:32.290729    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:32.559677 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:32.559653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7555-eb6312"]
Sep 10 15:01:32.561312 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:01:32.561284    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode887e08b_60a3_43fe_bd5c_2bd8a843f9b0.slice/crio-170185a8dae8d8f3aee0232ec967933732e7f469ba49058e66e1c729f4575b33 WatchSource:0}: Error finding container 170185a8dae8d8f3aee0232ec967933732e7f469ba49058e66e1c729f4575b33: Status 404 returned error can't find the container with id 170185a8dae8d8f3aee0232ec967933732e7f469ba49058e66e1c729f4575b33
Sep 10 15:01:32.913710 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:32.913673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7555-eb6312" event={"ID":"e887e08b-60a3-43fe-bd5c-2bd8a843f9b0","Type":"ContainerStarted","Data":"170185a8dae8d8f3aee0232ec967933732e7f469ba49058e66e1c729f4575b33"}
Sep 10 15:01:34.928436 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:34.928398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7555-eb6312" event={"ID":"e887e08b-60a3-43fe-bd5c-2bd8a843f9b0","Type":"ContainerStarted","Data":"ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634"}
Sep 10 15:01:34.945324 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:34.945280    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7555-eb6312" podStartSLOduration=2.211487612 podStartE2EDuration="3.945262659s" totalImagesPullingTime="1.733775047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:34.942826523 +0000 UTC m=+829.512365782" watchObservedRunningTime="2026-09-10 15:01:34.945262659 +0000 UTC m=+829.514801916"
Sep 10 15:01:35.552214 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:35.552178    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7555-eb6312"]
Sep 10 15:01:35.559209 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:35.559180    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7555-eb6312"]
Sep 10 15:01:35.934047 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:35.934015    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7555-eb6312" container="test-curl-7555-eb6312"
Sep 10 15:01:36.939093 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:36.939055    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7555-eb6312" podUID="e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" containerName="test-curl-7555-eb6312" containerID="cri-o://ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634" gracePeriod=2
Sep 10 15:01:39.122359 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.122332    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:39.296917 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.296829    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq96d\" (UniqueName: \"kubernetes.io/projected/e887e08b-60a3-43fe-bd5c-2bd8a843f9b0-kube-api-access-xq96d\") pod \"e887e08b-60a3-43fe-bd5c-2bd8a843f9b0\" (UID: \"e887e08b-60a3-43fe-bd5c-2bd8a843f9b0\") "
Sep 10 15:01:39.298936 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.298913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e887e08b-60a3-43fe-bd5c-2bd8a843f9b0-kube-api-access-xq96d" (OuterVolumeSpecName: "kube-api-access-xq96d") pod "e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" (UID: "e887e08b-60a3-43fe-bd5c-2bd8a843f9b0"). InnerVolumeSpecName "kube-api-access-xq96d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:01:39.398025 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.397987    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq96d\" (UniqueName: \"kubernetes.io/projected/e887e08b-60a3-43fe-bd5c-2bd8a843f9b0-kube-api-access-xq96d\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:01:39.954404 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.954365    2565 generic.go:356] "Generic (PLEG): container finished" podID="e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" containerID="ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634" exitCode=137
Sep 10 15:01:39.954597 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.954411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:39.954597 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.954447    2565 scope.go:117] "RemoveContainer" containerID="ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634"
Sep 10 15:01:39.963855 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.963834    2565 scope.go:117] "RemoveContainer" containerID="ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634"
Sep 10 15:01:39.964105 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:39.964087    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634\": container with ID starting with ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634 not found: ID does not exist" containerID="ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634"
Sep 10 15:01:39.964160 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.964112    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634"} err="failed to get container status \"ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634\": rpc error: code = NotFound desc = could not find container \"ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634\": container with ID starting with ca850a447dbe349eb53b2a25908a08a1cc06899a0260c6361ee1fd9da28af634 not found: ID does not exist"
Sep 10 15:01:39.964964 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:39.964940    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7555-eb6312\" is forbidden: User \"system:node:ip-10-0-130-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-2.ec2.internal' and this object" podUID="e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" pod="opendatahub/test-curl-7555-eb6312"
Sep 10 15:01:39.976187 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:39.976162    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" path="/var/lib/kubelet/pods/e887e08b-60a3-43fe-bd5c-2bd8a843f9b0/volumes"
Sep 10 15:01:40.827112 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.827082    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"]
Sep 10 15:01:40.827461 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.827415    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" containerName="test-curl-7555-eb6312"
Sep 10 15:01:40.827461 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.827426    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" containerName="test-curl-7555-eb6312"
Sep 10 15:01:40.827547 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.827489    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e887e08b-60a3-43fe-bd5c-2bd8a843f9b0" containerName="test-curl-7555-eb6312"
Sep 10 15:01:40.831857 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.831832    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:40.835052 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.835032    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c75e68-serving-cert\""
Sep 10 15:01:40.835486 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.835466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:01:40.843120 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.843101    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vh2ph\""
Sep 10 15:01:40.843491 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.843477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:01:40.843739 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.843712    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:01:40.861202 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:40.861172    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"]
Sep 10 15:01:41.010339 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.010306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhl5\" (UniqueName: \"kubernetes.io/projected/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-kube-api-access-6xhl5\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.010585 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.010347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.010585 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.010392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-metrics-certs\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.010585 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.010449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-openshift-service-ca\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.111775 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.111747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-metrics-certs\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.111918 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.111784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-openshift-service-ca\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.112049 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.112014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhl5\" (UniqueName: \"kubernetes.io/projected/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-kube-api-access-6xhl5\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.112257 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.112236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.112385 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:41.112362    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-serving-cert: secret "maas-api-e2e-autores-c75e68-serving-cert" not found
Sep 10 15:01:41.112462 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:41.112447    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls podName:47fd54b2-19ad-4ce5-9a32-9d1193a233d3 nodeName:}" failed. No retries permitted until 2026-09-10 15:01:41.612426961 +0000 UTC m=+836.181966218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls") pod "maas-api-e2e-autores-c75e68-f68c464b-2kntv" (UID: "47fd54b2-19ad-4ce5-9a32-9d1193a233d3") : secret "maas-api-e2e-autores-c75e68-serving-cert" not found
Sep 10 15:01:41.112532 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.112486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-openshift-service-ca\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.114271 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.114250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-metrics-certs\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.124261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.124240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhl5\" (UniqueName: \"kubernetes.io/projected/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-kube-api-access-6xhl5\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.617327 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.617299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.619699 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.619665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls\") pod \"maas-api-e2e-autores-c75e68-f68c464b-2kntv\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:41.741807 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:41.741763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:42.021481 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:42.021458    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"]
Sep 10 15:01:42.022768 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:01:42.022743    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fd54b2_19ad_4ce5_9a32_9d1193a233d3.slice/crio-dc633c22e305a10f457f5058a8db2682e2010ec170320a7aad3f24620324efc5 WatchSource:0}: Error finding container dc633c22e305a10f457f5058a8db2682e2010ec170320a7aad3f24620324efc5: Status 404 returned error can't find the container with id dc633c22e305a10f457f5058a8db2682e2010ec170320a7aad3f24620324efc5
Sep 10 15:01:42.975225 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:42.975163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv" event={"ID":"47fd54b2-19ad-4ce5-9a32-9d1193a233d3","Type":"ContainerStarted","Data":"dc633c22e305a10f457f5058a8db2682e2010ec170320a7aad3f24620324efc5"}
Sep 10 15:01:44.988345 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:44.988312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv" event={"ID":"47fd54b2-19ad-4ce5-9a32-9d1193a233d3","Type":"ContainerStarted","Data":"be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d"}
Sep 10 15:01:44.988767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:44.988435    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:45.017017 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:45.016968    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv" podStartSLOduration=2.744660414 podStartE2EDuration="5.016952042s" totalImagesPullingTime="2.272291628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:01:45.015055327 +0000 UTC m=+839.584594582" watchObservedRunningTime="2026-09-10 15:01:45.016952042 +0000 UTC m=+839.586491300"
Sep 10 15:01:45.078702 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:45.078663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:01:50.999707 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:50.999677    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:01:58.972371 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:58.972334    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"]
Sep 10 15:01:58.979209 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:58.979184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:58.982588 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:58.982562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6356e4-serving-cert\""
Sep 10 15:01:58.991576 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:58.991551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"]
Sep 10 15:01:59.070685 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.070630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4207e5-ca54-4847-b370-54e4fb327c76-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.070877 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.070733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kzj\" (UniqueName: \"kubernetes.io/projected/5a4207e5-ca54-4847-b370-54e4fb327c76-kube-api-access-62kzj\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.070877 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.070813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.070877 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.070843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-metrics-certs\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.172253 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.172222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4207e5-ca54-4847-b370-54e4fb327c76-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.172253 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.172261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kzj\" (UniqueName: \"kubernetes.io/projected/5a4207e5-ca54-4847-b370-54e4fb327c76-kube-api-access-62kzj\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.172514 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.172320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.172514 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.172351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-metrics-certs\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.172514 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:59.172454    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-serving-cert: secret "maas-api-e2e-inf-b-6356e4-serving-cert" not found
Sep 10 15:01:59.172694 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:59.172533    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls podName:5a4207e5-ca54-4847-b370-54e4fb327c76 nodeName:}" failed. No retries permitted until 2026-09-10 15:01:59.672515022 +0000 UTC m=+854.242054260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls") pod "maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" (UID: "5a4207e5-ca54-4847-b370-54e4fb327c76") : secret "maas-api-e2e-inf-b-6356e4-serving-cert" not found
Sep 10 15:01:59.172905 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.172884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4207e5-ca54-4847-b370-54e4fb327c76-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.174712 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.174692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-metrics-certs\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.183787 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.183762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kzj\" (UniqueName: \"kubernetes.io/projected/5a4207e5-ca54-4847-b370-54e4fb327c76-kube-api-access-62kzj\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.552322 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.552279    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:01:59.679030 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:01:59.678990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:01:59.679232 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:59.679098    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-serving-cert: secret "maas-api-e2e-inf-b-6356e4-serving-cert" not found
Sep 10 15:01:59.679232 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:01:59.679151    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls podName:5a4207e5-ca54-4847-b370-54e4fb327c76 nodeName:}" failed. No retries permitted until 2026-09-10 15:02:00.679137208 +0000 UTC m=+855.248676445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls") pod "maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" (UID: "5a4207e5-ca54-4847-b370-54e4fb327c76") : secret "maas-api-e2e-inf-b-6356e4-serving-cert" not found
Sep 10 15:02:00.688823 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:00.688785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:02:00.691207 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:00.691183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls\") pod \"maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:02:00.789922 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:00.789884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:02:01.067093 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:01.067070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"]
Sep 10 15:02:01.069340 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:02:01.069300    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4207e5_ca54_4847_b370_54e4fb327c76.slice/crio-7e3468a4fc4d8e96b3fea552727cbb94c68ef96e0e2cdd44795a43dda0c163d5 WatchSource:0}: Error finding container 7e3468a4fc4d8e96b3fea552727cbb94c68ef96e0e2cdd44795a43dda0c163d5: Status 404 returned error can't find the container with id 7e3468a4fc4d8e96b3fea552727cbb94c68ef96e0e2cdd44795a43dda0c163d5
Sep 10 15:02:01.079040 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:01.079013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" event={"ID":"5a4207e5-ca54-4847-b370-54e4fb327c76","Type":"ContainerStarted","Data":"7e3468a4fc4d8e96b3fea552727cbb94c68ef96e0e2cdd44795a43dda0c163d5"}
Sep 10 15:02:02.085599 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:02.085565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" event={"ID":"5a4207e5-ca54-4847-b370-54e4fb327c76","Type":"ContainerStarted","Data":"acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f"}
Sep 10 15:02:02.086021 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:02.085700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:02:02.106008 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:02.105960    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" podStartSLOduration=3.614834476 podStartE2EDuration="4.105946334s" totalImagesPullingTime="491.111858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:02.10385637 +0000 UTC m=+856.673395629" watchObservedRunningTime="2026-09-10 15:02:02.105946334 +0000 UTC m=+856.675485592"
Sep 10 15:02:08.096883 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:08.096857    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:02:13.553403 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.553363    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"]
Sep 10 15:02:13.556895 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.556878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.559126 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.559103    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-12631b8d-serving-cert\""
Sep 10 15:02:13.569093 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.569067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"]
Sep 10 15:02:13.709557 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.709530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-metrics-certs\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.709720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.709579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.709720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.709611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwgb\" (UniqueName: \"kubernetes.io/projected/08137c32-b498-4554-920c-d6d3ad20777b-kube-api-access-hwwgb\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.709720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.709680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08137c32-b498-4554-920c-d6d3ad20777b-openshift-service-ca\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.810417 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.810335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-metrics-certs\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.810417 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.810387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.810417 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.810408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwgb\" (UniqueName: \"kubernetes.io/projected/08137c32-b498-4554-920c-d6d3ad20777b-kube-api-access-hwwgb\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.810621 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.810474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08137c32-b498-4554-920c-d6d3ad20777b-openshift-service-ca\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.810621 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:02:13.810500    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-serving-cert: secret "maas-api-e2e-migrate-12631b8d-serving-cert" not found
Sep 10 15:02:13.810621 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:02:13.810589    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls podName:08137c32-b498-4554-920c-d6d3ad20777b nodeName:}" failed. No retries permitted until 2026-09-10 15:02:14.310559226 +0000 UTC m=+868.880098471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls") pod "maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6" (UID: "08137c32-b498-4554-920c-d6d3ad20777b") : secret "maas-api-e2e-migrate-12631b8d-serving-cert" not found
Sep 10 15:02:13.811180 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.811158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08137c32-b498-4554-920c-d6d3ad20777b-openshift-service-ca\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.812835 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.812813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-metrics-certs\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:13.821518 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:13.821496    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwgb\" (UniqueName: \"kubernetes.io/projected/08137c32-b498-4554-920c-d6d3ad20777b-kube-api-access-hwwgb\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:14.315728 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:14.315690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:14.318218 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:14.318185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls\") pod \"maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:14.468163 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:14.468123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:14.740081 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:14.740055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"]
Sep 10 15:02:14.741474 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:02:14.741450    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08137c32_b498_4554_920c_d6d3ad20777b.slice/crio-3bb2d514335eb1cdfc5af429c17da390f9d8bfbd57101feed89b8c2b06dcdef9 WatchSource:0}: Error finding container 3bb2d514335eb1cdfc5af429c17da390f9d8bfbd57101feed89b8c2b06dcdef9: Status 404 returned error can't find the container with id 3bb2d514335eb1cdfc5af429c17da390f9d8bfbd57101feed89b8c2b06dcdef9
Sep 10 15:02:15.156599 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:15.156561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6" event={"ID":"08137c32-b498-4554-920c-d6d3ad20777b","Type":"ContainerStarted","Data":"3bb2d514335eb1cdfc5af429c17da390f9d8bfbd57101feed89b8c2b06dcdef9"}
Sep 10 15:02:15.557835 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:15.557739    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:02:16.163417 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:16.163379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6" event={"ID":"08137c32-b498-4554-920c-d6d3ad20777b","Type":"ContainerStarted","Data":"59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e"}
Sep 10 15:02:16.163928 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:16.163485    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:16.179857 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:16.179810    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6" podStartSLOduration=2.708757969 podStartE2EDuration="3.179795431s" totalImagesPullingTime="471.037462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:16.179148317 +0000 UTC m=+870.748687569" watchObservedRunningTime="2026-09-10 15:02:16.179795431 +0000 UTC m=+870.749334689"
Sep 10 15:02:22.174455 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:22.174429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:27.857720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.857685    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"]
Sep 10 15:02:27.861362 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.861344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:27.863828 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.863804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-c1b5b19d-kserve-self-signed-certs\""
Sep 10 15:02:27.871538 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.871510    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"]
Sep 10 15:02:27.937871 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.937839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-home\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:27.938055 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.937875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-tmp-dir\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:27.938055 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.937956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vbn\" (UniqueName: \"kubernetes.io/projected/379c7f20-d197-4044-9afd-b3dec7ec86a6-kube-api-access-w4vbn\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:27.938055 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.938037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-dshm\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:27.938183 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.938062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-model-cache\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:27.938183 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:27.938088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379c7f20-d197-4044-9afd-b3dec7ec86a6-tls-certs\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-dshm\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-model-cache\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039386 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379c7f20-d197-4044-9afd-b3dec7ec86a6-tls-certs\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039386 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-home\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039386 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-tmp-dir\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039386 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vbn\" (UniqueName: \"kubernetes.io/projected/379c7f20-d197-4044-9afd-b3dec7ec86a6-kube-api-access-w4vbn\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039699 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-model-cache\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039797 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-tmp-dir\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.039993 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.039976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-home\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.042000 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.041979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-dshm\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.042272 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.042251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379c7f20-d197-4044-9afd-b3dec7ec86a6-tls-certs\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.048122 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.048098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vbn\" (UniqueName: \"kubernetes.io/projected/379c7f20-d197-4044-9afd-b3dec7ec86a6-kube-api-access-w4vbn\") pod \"e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") " pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.170179 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.170087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:28.445082 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:28.445052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"]
Sep 10 15:02:28.446163 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:02:28.446125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod379c7f20_d197_4044_9afd_b3dec7ec86a6.slice/crio-fe5316bfe1101cb2bc7737ef03c87ba035d72f79e67366acd30b44053091f85f WatchSource:0}: Error finding container fe5316bfe1101cb2bc7737ef03c87ba035d72f79e67366acd30b44053091f85f: Status 404 returned error can't find the container with id fe5316bfe1101cb2bc7737ef03c87ba035d72f79e67366acd30b44053091f85f
Sep 10 15:02:29.235129 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:29.235093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq" event={"ID":"379c7f20-d197-4044-9afd-b3dec7ec86a6","Type":"ContainerStarted","Data":"b4292333e5d721c9d6519f00daa2d73a517ae9ab90cbdbabf32825450252f526"}
Sep 10 15:02:29.235530 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:29.235143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq" event={"ID":"379c7f20-d197-4044-9afd-b3dec7ec86a6","Type":"ContainerStarted","Data":"fe5316bfe1101cb2bc7737ef03c87ba035d72f79e67366acd30b44053091f85f"}
Sep 10 15:02:29.254541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:29.254488    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq" podStartSLOduration=2.25447357 podStartE2EDuration="2.25447357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:29.252357619 +0000 UTC m=+883.821896878" watchObservedRunningTime="2026-09-10 15:02:29.25447357 +0000 UTC m=+883.824012828"
Sep 10 15:02:31.056141 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:31.056101    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:02:38.170951 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:38.170917    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:38.171369 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:38.170967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:38.183783 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:38.183750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:38.299036 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:38.299010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:02:44.156285 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.156251    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"]
Sep 10 15:02:44.159774 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.159753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.162996 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.162975    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c75e68\"/\"openshift-service-ca.crt\""
Sep 10 15:02:44.163111 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.163001    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c75e68\"/\"default-dockercfg-r4269\""
Sep 10 15:02:44.163849 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.163831    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c75e68\"/\"autores-model-c75e68-kserve-self-signed-certs\""
Sep 10 15:02:44.163938 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.163889    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c75e68\"/\"kube-root-ca.crt\""
Sep 10 15:02:44.169855 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.169830    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"]
Sep 10 15:02:44.289848 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.289810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7815f442-9c94-424f-baef-a8211d12c24d-tls-certs\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.289848 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.289855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-home\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.290075 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.289881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-dshm\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.290075 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.289983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bcz\" (UniqueName: \"kubernetes.io/projected/7815f442-9c94-424f-baef-a8211d12c24d-kube-api-access-56bcz\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.290075 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.290029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-tmp-dir\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.290075 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.290060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-model-cache\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.390878 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.390842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7815f442-9c94-424f-baef-a8211d12c24d-tls-certs\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.390878 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.390881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-home\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.391076 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.390903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-dshm\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.391076 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.391018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bcz\" (UniqueName: \"kubernetes.io/projected/7815f442-9c94-424f-baef-a8211d12c24d-kube-api-access-56bcz\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.391158 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.391103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-tmp-dir\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.391158 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.391136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-model-cache\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.391271 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.391221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-home\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.391496 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.391476    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-model-cache\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.391496 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.391478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-tmp-dir\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.393171 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.393151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-dshm\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.393408 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.393393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7815f442-9c94-424f-baef-a8211d12c24d-tls-certs\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.399552 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.399531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bcz\" (UniqueName: \"kubernetes.io/projected/7815f442-9c94-424f-baef-a8211d12c24d-kube-api-access-56bcz\") pod \"autores-model-c75e68-kserve-55544c6b87-27skc\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") " pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.469564 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.469494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:44.742200 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:44.742174    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"]
Sep 10 15:02:44.744096 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:02:44.744068    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7815f442_9c94_424f_baef_a8211d12c24d.slice/crio-aad1b02aed95adb3b6b45b6d396207b2978dcd859eaad9c4b65e24bc7d22adac WatchSource:0}: Error finding container aad1b02aed95adb3b6b45b6d396207b2978dcd859eaad9c4b65e24bc7d22adac: Status 404 returned error can't find the container with id aad1b02aed95adb3b6b45b6d396207b2978dcd859eaad9c4b65e24bc7d22adac
Sep 10 15:02:45.079958 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.079865    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"]
Sep 10 15:02:45.080118 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.080066    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6" podUID="08137c32-b498-4554-920c-d6d3ad20777b" containerName="maas-api" containerID="cri-o://59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e" gracePeriod=30
Sep 10 15:02:45.313526 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.313501    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:45.319696 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.319666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc" event={"ID":"7815f442-9c94-424f-baef-a8211d12c24d","Type":"ContainerStarted","Data":"dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028"}
Sep 10 15:02:45.319840 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.319713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc" event={"ID":"7815f442-9c94-424f-baef-a8211d12c24d","Type":"ContainerStarted","Data":"aad1b02aed95adb3b6b45b6d396207b2978dcd859eaad9c4b65e24bc7d22adac"}
Sep 10 15:02:45.320999 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.320979    2565 generic.go:356] "Generic (PLEG): container finished" podID="08137c32-b498-4554-920c-d6d3ad20777b" containerID="59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e" exitCode=0
Sep 10 15:02:45.321092 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.321006    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"
Sep 10 15:02:45.321092 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.321026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6" event={"ID":"08137c32-b498-4554-920c-d6d3ad20777b","Type":"ContainerDied","Data":"59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e"}
Sep 10 15:02:45.321092 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.321063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6" event={"ID":"08137c32-b498-4554-920c-d6d3ad20777b","Type":"ContainerDied","Data":"3bb2d514335eb1cdfc5af429c17da390f9d8bfbd57101feed89b8c2b06dcdef9"}
Sep 10 15:02:45.321092 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.321084    2565 scope.go:117] "RemoveContainer" containerID="59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e"
Sep 10 15:02:45.329429 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.329409    2565 scope.go:117] "RemoveContainer" containerID="59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e"
Sep 10 15:02:45.329701 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:02:45.329679    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e\": container with ID starting with 59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e not found: ID does not exist" containerID="59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e"
Sep 10 15:02:45.329767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.329709    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e"} err="failed to get container status \"59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e\": rpc error: code = NotFound desc = could not find container \"59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e\": container with ID starting with 59e411548c1c5f27fbbb315cc62cd712503520c43fb3d8c0bc67c7b957705e6e not found: ID does not exist"
Sep 10 15:02:45.362020 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.361964    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc" podStartSLOduration=1.361945974 podStartE2EDuration="1.361945974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:02:45.357287763 +0000 UTC m=+899.926827023" watchObservedRunningTime="2026-09-10 15:02:45.361945974 +0000 UTC m=+899.931485233"
Sep 10 15:02:45.400210 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.400173    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08137c32-b498-4554-920c-d6d3ad20777b-openshift-service-ca\") pod \"08137c32-b498-4554-920c-d6d3ad20777b\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") "
Sep 10 15:02:45.400317 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.400259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwwgb\" (UniqueName: \"kubernetes.io/projected/08137c32-b498-4554-920c-d6d3ad20777b-kube-api-access-hwwgb\") pod \"08137c32-b498-4554-920c-d6d3ad20777b\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") "
Sep 10 15:02:45.400317 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.400306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-metrics-certs\") pod \"08137c32-b498-4554-920c-d6d3ad20777b\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") "
Sep 10 15:02:45.400424 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.400334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls\") pod \"08137c32-b498-4554-920c-d6d3ad20777b\" (UID: \"08137c32-b498-4554-920c-d6d3ad20777b\") "
Sep 10 15:02:45.400618 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.400593    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08137c32-b498-4554-920c-d6d3ad20777b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08137c32-b498-4554-920c-d6d3ad20777b" (UID: "08137c32-b498-4554-920c-d6d3ad20777b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:02:45.400794 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.400767    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08137c32-b498-4554-920c-d6d3ad20777b-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:02:45.402767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.402740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08137c32-b498-4554-920c-d6d3ad20777b" (UID: "08137c32-b498-4554-920c-d6d3ad20777b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:02:45.402898 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.402873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "08137c32-b498-4554-920c-d6d3ad20777b" (UID: "08137c32-b498-4554-920c-d6d3ad20777b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:02:45.402943 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.402883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08137c32-b498-4554-920c-d6d3ad20777b-kube-api-access-hwwgb" (OuterVolumeSpecName: "kube-api-access-hwwgb") pod "08137c32-b498-4554-920c-d6d3ad20777b" (UID: "08137c32-b498-4554-920c-d6d3ad20777b"). InnerVolumeSpecName "kube-api-access-hwwgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:02:45.502318 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.502284    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:02:45.502318 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.502321    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08137c32-b498-4554-920c-d6d3ad20777b-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:02:45.502584 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.502334    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwwgb\" (UniqueName: \"kubernetes.io/projected/08137c32-b498-4554-920c-d6d3ad20777b-kube-api-access-hwwgb\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:02:45.660615 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.660585    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"]
Sep 10 15:02:45.673710 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.673685    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-12631b8d-8f4748b47-pv5d6"]
Sep 10 15:02:45.976991 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:45.976950    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08137c32-b498-4554-920c-d6d3ad20777b" path="/var/lib/kubelet/pods/08137c32-b498-4554-920c-d6d3ad20777b/volumes"
Sep 10 15:02:50.658885 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:50.658844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:02:54.470258 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:54.470213    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:54.470711 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:54.470273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:54.482770 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:54.482747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:02:55.385620 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:02:55.385583    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:03:07.036867 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.036828    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"]
Sep 10 15:03:07.037344 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.037208    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08137c32-b498-4554-920c-d6d3ad20777b" containerName="maas-api"
Sep 10 15:03:07.037344 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.037220    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08137c32-b498-4554-920c-d6d3ad20777b" containerName="maas-api"
Sep 10 15:03:07.037344 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.037280    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08137c32-b498-4554-920c-d6d3ad20777b" containerName="maas-api"
Sep 10 15:03:07.040633 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.040617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.044034 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.044010    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e577f0\"/\"default-dockercfg-w6p86\""
Sep 10 15:03:07.044155 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.044035    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e577f0\"/\"kube-root-ca.crt\""
Sep 10 15:03:07.044155 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.044013    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e577f0\"/\"openshift-service-ca.crt\""
Sep 10 15:03:07.044155 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.044010    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e577f0\"/\"test-model-e577f0-kserve-self-signed-certs\""
Sep 10 15:03:07.050919 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.050896    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"]
Sep 10 15:03:07.216737 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.216701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb53c36-c341-4fea-90df-eceb3e063e14-tls-certs\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.216926 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.216752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-home\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.216926 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.216824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-tmp-dir\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.216926 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.216868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2gh\" (UniqueName: \"kubernetes.io/projected/9eb53c36-c341-4fea-90df-eceb3e063e14-kube-api-access-th2gh\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.217058 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.216935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-dshm\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.217058 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.216963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-model-cache\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.318507 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.318413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb53c36-c341-4fea-90df-eceb3e063e14-tls-certs\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.318507 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.318464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-home\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.318507 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.318501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-tmp-dir\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.318824 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.318522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2gh\" (UniqueName: \"kubernetes.io/projected/9eb53c36-c341-4fea-90df-eceb3e063e14-kube-api-access-th2gh\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.318824 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.318559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-dshm\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.318824 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.318583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-model-cache\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.319061 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.319035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-model-cache\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.319061 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.319049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-tmp-dir\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.319130 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.319103    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-home\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.320833 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.320807    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-dshm\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.321106 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.321089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb53c36-c341-4fea-90df-eceb3e063e14-tls-certs\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.327751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.327727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2gh\" (UniqueName: \"kubernetes.io/projected/9eb53c36-c341-4fea-90df-eceb3e063e14-kube-api-access-th2gh\") pod \"test-model-e577f0-kserve-757fff4dd5-w79ng\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") " pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.350654 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.350619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:07.624799 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:07.624773    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"]
Sep 10 15:03:07.626999 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:03:07.626972    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eb53c36_c341_4fea_90df_eceb3e063e14.slice/crio-e979e1f7f153e0b700c42fbd4b4c9e7b31f2fc4969c5e96b964c1cc27c3cb38a WatchSource:0}: Error finding container e979e1f7f153e0b700c42fbd4b4c9e7b31f2fc4969c5e96b964c1cc27c3cb38a: Status 404 returned error can't find the container with id e979e1f7f153e0b700c42fbd4b4c9e7b31f2fc4969c5e96b964c1cc27c3cb38a
Sep 10 15:03:08.449096 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:08.449056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng" event={"ID":"9eb53c36-c341-4fea-90df-eceb3e063e14","Type":"ContainerStarted","Data":"a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374"}
Sep 10 15:03:08.449508 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:08.449108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng" event={"ID":"9eb53c36-c341-4fea-90df-eceb3e063e14","Type":"ContainerStarted","Data":"e979e1f7f153e0b700c42fbd4b4c9e7b31f2fc4969c5e96b964c1cc27c3cb38a"}
Sep 10 15:03:08.468238 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:08.468184    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng" podStartSLOduration=1.4681686919999999 podStartE2EDuration="1.468168692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:08.467247992 +0000 UTC m=+923.036787271" watchObservedRunningTime="2026-09-10 15:03:08.468168692 +0000 UTC m=+923.037707966"
Sep 10 15:03:11.778216 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:11.778184    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:03:17.351130 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:17.351095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:17.351497 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:17.351139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:17.367512 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:17.367476    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:17.509134 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:17.509097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:03:27.669110 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:27.669074    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"]
Sep 10 15:03:27.669562 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:27.669315    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv" podUID="47fd54b2-19ad-4ce5-9a32-9d1193a233d3" containerName="maas-api" containerID="cri-o://be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d" gracePeriod=30
Sep 10 15:03:27.920455 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:27.920394    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:03:28.015873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.015838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xhl5\" (UniqueName: \"kubernetes.io/projected/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-kube-api-access-6xhl5\") pod \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") "
Sep 10 15:03:28.016039 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.015883    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls\") pod \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") "
Sep 10 15:03:28.016039 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.015899    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-openshift-service-ca\") pod \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") "
Sep 10 15:03:28.016039 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.015945    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-metrics-certs\") pod \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\" (UID: \"47fd54b2-19ad-4ce5-9a32-9d1193a233d3\") "
Sep 10 15:03:28.016281 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.016252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47fd54b2-19ad-4ce5-9a32-9d1193a233d3" (UID: "47fd54b2-19ad-4ce5-9a32-9d1193a233d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:03:28.018277 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.018242    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "47fd54b2-19ad-4ce5-9a32-9d1193a233d3" (UID: "47fd54b2-19ad-4ce5-9a32-9d1193a233d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:03:28.018413 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.018326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47fd54b2-19ad-4ce5-9a32-9d1193a233d3" (UID: "47fd54b2-19ad-4ce5-9a32-9d1193a233d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:03:28.018413 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.018376    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-kube-api-access-6xhl5" (OuterVolumeSpecName: "kube-api-access-6xhl5") pod "47fd54b2-19ad-4ce5-9a32-9d1193a233d3" (UID: "47fd54b2-19ad-4ce5-9a32-9d1193a233d3"). InnerVolumeSpecName "kube-api-access-6xhl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:03:28.116959 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.116926    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:28.116959 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.116952    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xhl5\" (UniqueName: \"kubernetes.io/projected/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-kube-api-access-6xhl5\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:28.116959 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.116963    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:28.117189 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.116972    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47fd54b2-19ad-4ce5-9a32-9d1193a233d3-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:28.565241 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.565202    2565 generic.go:356] "Generic (PLEG): container finished" podID="47fd54b2-19ad-4ce5-9a32-9d1193a233d3" containerID="be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d" exitCode=0
Sep 10 15:03:28.565422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.565258    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"
Sep 10 15:03:28.565422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.565283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv" event={"ID":"47fd54b2-19ad-4ce5-9a32-9d1193a233d3","Type":"ContainerDied","Data":"be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d"}
Sep 10 15:03:28.565422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.565357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv" event={"ID":"47fd54b2-19ad-4ce5-9a32-9d1193a233d3","Type":"ContainerDied","Data":"dc633c22e305a10f457f5058a8db2682e2010ec170320a7aad3f24620324efc5"}
Sep 10 15:03:28.565422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.565379    2565 scope.go:117] "RemoveContainer" containerID="be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d"
Sep 10 15:03:28.574966 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.574950    2565 scope.go:117] "RemoveContainer" containerID="be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d"
Sep 10 15:03:28.575262 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:03:28.575242    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d\": container with ID starting with be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d not found: ID does not exist" containerID="be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d"
Sep 10 15:03:28.575329 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.575271    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d"} err="failed to get container status \"be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d\": rpc error: code = NotFound desc = could not find container \"be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d\": container with ID starting with be5a24d7e403c32dcde7a3441ae57ef3085513bc263f3bb90face76fa21ba72d not found: ID does not exist"
Sep 10 15:03:28.587786 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.587761    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"]
Sep 10 15:03:28.590065 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:28.590044    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c75e68-f68c464b-2kntv"]
Sep 10 15:03:29.977464 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:29.977423    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47fd54b2-19ad-4ce5-9a32-9d1193a233d3" path="/var/lib/kubelet/pods/47fd54b2-19ad-4ce5-9a32-9d1193a233d3/volumes"
Sep 10 15:03:33.252474 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:33.252440    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"]
Sep 10 15:03:34.685905 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:34.685864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:03:46.493549 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.493515    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"]
Sep 10 15:03:46.493970 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.493888    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fd54b2-19ad-4ce5-9a32-9d1193a233d3" containerName="maas-api"
Sep 10 15:03:46.493970 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.493901    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fd54b2-19ad-4ce5-9a32-9d1193a233d3" containerName="maas-api"
Sep 10 15:03:46.494049 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.493998    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fd54b2-19ad-4ce5-9a32-9d1193a233d3" containerName="maas-api"
Sep 10 15:03:46.498168 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.498150    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.500809 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.500789    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-c1b5b19d-kserve-self-signed-certs\""
Sep 10 15:03:46.515649 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.515616    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"]
Sep 10 15:03:46.695611 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.695577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn7b\" (UniqueName: \"kubernetes.io/projected/62f827e6-6126-494e-b162-7b513bfab77e-kube-api-access-pmn7b\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.695611 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.695612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-tmp-dir\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.695844 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.695635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f827e6-6126-494e-b162-7b513bfab77e-tls-certs\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.695844 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.695673    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-model-cache\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.695844 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.695740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-home\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.695844 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.695779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-dshm\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.796945 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.796862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn7b\" (UniqueName: \"kubernetes.io/projected/62f827e6-6126-494e-b162-7b513bfab77e-kube-api-access-pmn7b\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.796945 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.796898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-tmp-dir\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.796945 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.796924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f827e6-6126-494e-b162-7b513bfab77e-tls-certs\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.796945 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.796947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-model-cache\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.797277 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.796981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-home\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.797277 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.797025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-dshm\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.797391 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.797340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-tmp-dir\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.797444 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.797410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-model-cache\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.797480 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.797413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-home\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.799370 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.799348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-dshm\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.799501 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.799481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f827e6-6126-494e-b162-7b513bfab77e-tls-certs\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.805941 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.805920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn7b\" (UniqueName: \"kubernetes.io/projected/62f827e6-6126-494e-b162-7b513bfab77e-kube-api-access-pmn7b\") pod \"e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") " pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:46.806779 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:46.806749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:47.090080 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:47.090048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"]
Sep 10 15:03:47.091551 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:03:47.091516    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62f827e6_6126_494e_b162_7b513bfab77e.slice/crio-7f5c22f801a2fe42f03ba4d8dbe978c12886fc7c62c1c03834c20c8d00967510 WatchSource:0}: Error finding container 7f5c22f801a2fe42f03ba4d8dbe978c12886fc7c62c1c03834c20c8d00967510: Status 404 returned error can't find the container with id 7f5c22f801a2fe42f03ba4d8dbe978c12886fc7c62c1c03834c20c8d00967510
Sep 10 15:03:47.668934 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:47.668895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl" event={"ID":"62f827e6-6126-494e-b162-7b513bfab77e","Type":"ContainerStarted","Data":"8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309"}
Sep 10 15:03:47.668934 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:47.668939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl" event={"ID":"62f827e6-6126-494e-b162-7b513bfab77e","Type":"ContainerStarted","Data":"7f5c22f801a2fe42f03ba4d8dbe978c12886fc7c62c1c03834c20c8d00967510"}
Sep 10 15:03:47.691714 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:47.691666    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl" podStartSLOduration=1.6916509180000001 podStartE2EDuration="1.691650918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:47.691447923 +0000 UTC m=+962.260987180" watchObservedRunningTime="2026-09-10 15:03:47.691650918 +0000 UTC m=+962.261190167"
Sep 10 15:03:48.272284 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.272246    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc" podUID="7815f442-9c94-424f-baef-a8211d12c24d" containerName="main" containerID="cri-o://dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028" gracePeriod=45
Sep 10 15:03:48.521476 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.521451    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:03:48.611829 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.611799    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-home\") pod \"7815f442-9c94-424f-baef-a8211d12c24d\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") "
Sep 10 15:03:48.611995 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.611844    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56bcz\" (UniqueName: \"kubernetes.io/projected/7815f442-9c94-424f-baef-a8211d12c24d-kube-api-access-56bcz\") pod \"7815f442-9c94-424f-baef-a8211d12c24d\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") "
Sep 10 15:03:48.611995 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.611872    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7815f442-9c94-424f-baef-a8211d12c24d-tls-certs\") pod \"7815f442-9c94-424f-baef-a8211d12c24d\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") "
Sep 10 15:03:48.611995 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.611912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-tmp-dir\") pod \"7815f442-9c94-424f-baef-a8211d12c24d\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") "
Sep 10 15:03:48.611995 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.611939    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-model-cache\") pod \"7815f442-9c94-424f-baef-a8211d12c24d\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") "
Sep 10 15:03:48.612218 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.612030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-dshm\") pod \"7815f442-9c94-424f-baef-a8211d12c24d\" (UID: \"7815f442-9c94-424f-baef-a8211d12c24d\") "
Sep 10 15:03:48.612218 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.612085    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-home" (OuterVolumeSpecName: "home") pod "7815f442-9c94-424f-baef-a8211d12c24d" (UID: "7815f442-9c94-424f-baef-a8211d12c24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:03:48.612218 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.612161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7815f442-9c94-424f-baef-a8211d12c24d" (UID: "7815f442-9c94-424f-baef-a8211d12c24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:03:48.612358 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.612250    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7815f442-9c94-424f-baef-a8211d12c24d" (UID: "7815f442-9c94-424f-baef-a8211d12c24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:03:48.612398 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.612356    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:48.612398 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.612376    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:48.612398 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.612389    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:48.614005 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.613983    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-dshm" (OuterVolumeSpecName: "dshm") pod "7815f442-9c94-424f-baef-a8211d12c24d" (UID: "7815f442-9c94-424f-baef-a8211d12c24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:03:48.614005 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.614000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7815f442-9c94-424f-baef-a8211d12c24d-kube-api-access-56bcz" (OuterVolumeSpecName: "kube-api-access-56bcz") pod "7815f442-9c94-424f-baef-a8211d12c24d" (UID: "7815f442-9c94-424f-baef-a8211d12c24d"). InnerVolumeSpecName "kube-api-access-56bcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:03:48.614128 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.613985    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7815f442-9c94-424f-baef-a8211d12c24d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7815f442-9c94-424f-baef-a8211d12c24d" (UID: "7815f442-9c94-424f-baef-a8211d12c24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:03:48.675448 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.675412    2565 generic.go:356] "Generic (PLEG): container finished" podID="7815f442-9c94-424f-baef-a8211d12c24d" containerID="dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028" exitCode=0
Sep 10 15:03:48.675849 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.675459    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"
Sep 10 15:03:48.675849 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.675494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc" event={"ID":"7815f442-9c94-424f-baef-a8211d12c24d","Type":"ContainerDied","Data":"dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028"}
Sep 10 15:03:48.675849 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.675548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc" event={"ID":"7815f442-9c94-424f-baef-a8211d12c24d","Type":"ContainerDied","Data":"aad1b02aed95adb3b6b45b6d396207b2978dcd859eaad9c4b65e24bc7d22adac"}
Sep 10 15:03:48.675849 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.675567    2565 scope.go:117] "RemoveContainer" containerID="dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028"
Sep 10 15:03:48.684779 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.684759    2565 scope.go:117] "RemoveContainer" containerID="dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028"
Sep 10 15:03:48.685057 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:03:48.685038    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028\": container with ID starting with dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028 not found: ID does not exist" containerID="dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028"
Sep 10 15:03:48.685110 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.685064    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028"} err="failed to get container status \"dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028\": rpc error: code = NotFound desc = could not find container \"dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028\": container with ID starting with dd82639ec045e035602f212887011009d763dbf028faa2b6abac5bb789db8028 not found: ID does not exist"
Sep 10 15:03:48.703209 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.703182    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"]
Sep 10 15:03:48.705422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.705403    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c75e68/autores-model-c75e68-kserve-55544c6b87-27skc"]
Sep 10 15:03:48.712803 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.712785    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7815f442-9c94-424f-baef-a8211d12c24d-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:48.712863 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.712808    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bcz\" (UniqueName: \"kubernetes.io/projected/7815f442-9c94-424f-baef-a8211d12c24d-kube-api-access-56bcz\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:48.712863 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:48.712822    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7815f442-9c94-424f-baef-a8211d12c24d-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:03:49.977635 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:49.977602    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7815f442-9c94-424f-baef-a8211d12c24d" path="/var/lib/kubelet/pods/7815f442-9c94-424f-baef-a8211d12c24d/volumes"
Sep 10 15:03:53.579739 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:53.579698    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:03:56.807462 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:56.807426    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:56.807864 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:56.807508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:56.820520 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:56.820472    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:03:57.736795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:03:57.736767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:04:09.174281 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:09.174201    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:04:10.518627 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:10.518591    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"]
Sep 10 15:04:12.725233 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:12.725199    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"]
Sep 10 15:04:17.894576 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.894536    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"]
Sep 10 15:04:17.895081 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.894932    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7815f442-9c94-424f-baef-a8211d12c24d" containerName="main"
Sep 10 15:04:17.895081 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.894945    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7815f442-9c94-424f-baef-a8211d12c24d" containerName="main"
Sep 10 15:04:17.895081 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.895015    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7815f442-9c94-424f-baef-a8211d12c24d" containerName="main"
Sep 10 15:04:17.898197 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.898177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:17.901299 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.901273    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6356e4\"/\"kube-root-ca.crt\""
Sep 10 15:04:17.902269 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.902251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6356e4\"/\"test-model-6356e4-kserve-self-signed-certs\""
Sep 10 15:04:17.902269 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.902260    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6356e4\"/\"default-dockercfg-s8h6p\""
Sep 10 15:04:17.902390 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.902340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6356e4\"/\"openshift-service-ca.crt\""
Sep 10 15:04:17.910606 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.910581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"]
Sep 10 15:04:17.983565 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.983537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-dshm\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:17.983770 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.983584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tmp-dir\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:17.983770 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.983605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-home\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:17.983770 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.983624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-model-cache\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:17.983770 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.983692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnfb\" (UniqueName: \"kubernetes.io/projected/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-kube-api-access-qcnfb\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:17.983770 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:17.983717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tls-certs\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.084933 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.084892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-dshm\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085113 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.084946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tmp-dir\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085113 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.085060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-home\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085113 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.085097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-model-cache\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085284 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.085219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnfb\" (UniqueName: \"kubernetes.io/projected/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-kube-api-access-qcnfb\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085284 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.085257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tmp-dir\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085394 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.085288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tls-certs\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085394 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.085374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-model-cache\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.085500 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.085446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-home\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.087311 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.087282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-dshm\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.087592 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.087574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tls-certs\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.094075 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.094051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnfb\" (UniqueName: \"kubernetes.io/projected/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-kube-api-access-qcnfb\") pod \"test-model-6356e4-kserve-794dbb9c6c-d4f55\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") " pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.208624 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.208538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:18.485384 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.485360    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"]
Sep 10 15:04:18.487217 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:04:18.487191    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc27e2f98_9e31_4f2c_a939_7a3fcd10a031.slice/crio-e89fced963db76146f4b19a8146effcd357f9c786c20c7debac70c7d3802a645 WatchSource:0}: Error finding container e89fced963db76146f4b19a8146effcd357f9c786c20c7debac70c7d3802a645: Status 404 returned error can't find the container with id e89fced963db76146f4b19a8146effcd357f9c786c20c7debac70c7d3802a645
Sep 10 15:04:18.838055 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.837968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55" event={"ID":"c27e2f98-9e31-4f2c-a939-7a3fcd10a031","Type":"ContainerStarted","Data":"232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7"}
Sep 10 15:04:18.838055 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.838016    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55" event={"ID":"c27e2f98-9e31-4f2c-a939-7a3fcd10a031","Type":"ContainerStarted","Data":"e89fced963db76146f4b19a8146effcd357f9c786c20c7debac70c7d3802a645"}
Sep 10 15:04:18.859869 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:18.859818    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55" podStartSLOduration=1.859803255 podStartE2EDuration="1.859803255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:04:18.858961226 +0000 UTC m=+993.428500496" watchObservedRunningTime="2026-09-10 15:04:18.859803255 +0000 UTC m=+993.429342514"
Sep 10 15:04:24.971423 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:24.971385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:04:25.539510 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.539468    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl" podUID="62f827e6-6126-494e-b162-7b513bfab77e" containerName="main" containerID="cri-o://8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309" gracePeriod=45
Sep 10 15:04:25.782200 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.782171    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:04:25.875538 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.875495    2565 generic.go:356] "Generic (PLEG): container finished" podID="62f827e6-6126-494e-b162-7b513bfab77e" containerID="8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309" exitCode=0
Sep 10 15:04:25.875719 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.875547    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"
Sep 10 15:04:25.875719 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.875580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl" event={"ID":"62f827e6-6126-494e-b162-7b513bfab77e","Type":"ContainerDied","Data":"8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309"}
Sep 10 15:04:25.875719 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.875616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl" event={"ID":"62f827e6-6126-494e-b162-7b513bfab77e","Type":"ContainerDied","Data":"7f5c22f801a2fe42f03ba4d8dbe978c12886fc7c62c1c03834c20c8d00967510"}
Sep 10 15:04:25.875719 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.875633    2565 scope.go:117] "RemoveContainer" containerID="8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309"
Sep 10 15:04:25.884316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.884301    2565 scope.go:117] "RemoveContainer" containerID="8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309"
Sep 10 15:04:25.884560 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:04:25.884542    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309\": container with ID starting with 8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309 not found: ID does not exist" containerID="8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309"
Sep 10 15:04:25.884605 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.884568    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309"} err="failed to get container status \"8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309\": rpc error: code = NotFound desc = could not find container \"8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309\": container with ID starting with 8ff3e4dccc2bffe7e4c37fa1968bfe8861233e8b6cb0c7bd8f91d2383966d309 not found: ID does not exist"
Sep 10 15:04:25.962356 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962322    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-dshm\") pod \"62f827e6-6126-494e-b162-7b513bfab77e\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") "
Sep 10 15:04:25.962506 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmn7b\" (UniqueName: \"kubernetes.io/projected/62f827e6-6126-494e-b162-7b513bfab77e-kube-api-access-pmn7b\") pod \"62f827e6-6126-494e-b162-7b513bfab77e\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") "
Sep 10 15:04:25.962506 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962406    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-model-cache\") pod \"62f827e6-6126-494e-b162-7b513bfab77e\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") "
Sep 10 15:04:25.962506 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-tmp-dir\") pod \"62f827e6-6126-494e-b162-7b513bfab77e\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") "
Sep 10 15:04:25.962506 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962502    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-home\") pod \"62f827e6-6126-494e-b162-7b513bfab77e\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") "
Sep 10 15:04:25.962724 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f827e6-6126-494e-b162-7b513bfab77e-tls-certs\") pod \"62f827e6-6126-494e-b162-7b513bfab77e\" (UID: \"62f827e6-6126-494e-b162-7b513bfab77e\") "
Sep 10 15:04:25.962781 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962756    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-model-cache" (OuterVolumeSpecName: "model-cache") pod "62f827e6-6126-494e-b162-7b513bfab77e" (UID: "62f827e6-6126-494e-b162-7b513bfab77e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:25.962859 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962831    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62f827e6-6126-494e-b162-7b513bfab77e" (UID: "62f827e6-6126-494e-b162-7b513bfab77e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:25.962932 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962848    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-home" (OuterVolumeSpecName: "home") pod "62f827e6-6126-494e-b162-7b513bfab77e" (UID: "62f827e6-6126-494e-b162-7b513bfab77e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:25.962989 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962961    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:25.962989 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962980    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:25.963058 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.962991    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:25.964673 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.964628    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-dshm" (OuterVolumeSpecName: "dshm") pod "62f827e6-6126-494e-b162-7b513bfab77e" (UID: "62f827e6-6126-494e-b162-7b513bfab77e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:25.964673 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.964666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f827e6-6126-494e-b162-7b513bfab77e-kube-api-access-pmn7b" (OuterVolumeSpecName: "kube-api-access-pmn7b") pod "62f827e6-6126-494e-b162-7b513bfab77e" (UID: "62f827e6-6126-494e-b162-7b513bfab77e"). InnerVolumeSpecName "kube-api-access-pmn7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:25.964827 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:25.964681    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f827e6-6126-494e-b162-7b513bfab77e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62f827e6-6126-494e-b162-7b513bfab77e" (UID: "62f827e6-6126-494e-b162-7b513bfab77e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:26.064390 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:26.064365    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f827e6-6126-494e-b162-7b513bfab77e-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:26.064390 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:26.064389    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f827e6-6126-494e-b162-7b513bfab77e-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:26.064789 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:26.064398    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmn7b\" (UniqueName: \"kubernetes.io/projected/62f827e6-6126-494e-b162-7b513bfab77e-kube-api-access-pmn7b\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:26.193316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:26.193236    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"]
Sep 10 15:04:26.197709 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:26.197687    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-c1b5b19d-kserve-797564777b-4v9dl"]
Sep 10 15:04:27.744353 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:27.744315    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq" podUID="379c7f20-d197-4044-9afd-b3dec7ec86a6" containerName="main" containerID="cri-o://b4292333e5d721c9d6519f00daa2d73a517ae9ab90cbdbabf32825450252f526" gracePeriod=45
Sep 10 15:04:27.890064 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:27.890033    2565 generic.go:356] "Generic (PLEG): container finished" podID="379c7f20-d197-4044-9afd-b3dec7ec86a6" containerID="b4292333e5d721c9d6519f00daa2d73a517ae9ab90cbdbabf32825450252f526" exitCode=0
Sep 10 15:04:27.890232 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:27.890103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq" event={"ID":"379c7f20-d197-4044-9afd-b3dec7ec86a6","Type":"ContainerDied","Data":"b4292333e5d721c9d6519f00daa2d73a517ae9ab90cbdbabf32825450252f526"}
Sep 10 15:04:27.977410 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:27.977380    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f827e6-6126-494e-b162-7b513bfab77e" path="/var/lib/kubelet/pods/62f827e6-6126-494e-b162-7b513bfab77e/volumes"
Sep 10 15:04:27.994875 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:27.994815    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:04:28.183549 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.183501    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4vbn\" (UniqueName: \"kubernetes.io/projected/379c7f20-d197-4044-9afd-b3dec7ec86a6-kube-api-access-w4vbn\") pod \"379c7f20-d197-4044-9afd-b3dec7ec86a6\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") "
Sep 10 15:04:28.183760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.183583    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-dshm\") pod \"379c7f20-d197-4044-9afd-b3dec7ec86a6\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") "
Sep 10 15:04:28.183760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.183604    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-tmp-dir\") pod \"379c7f20-d197-4044-9afd-b3dec7ec86a6\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") "
Sep 10 15:04:28.183760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.183634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-home\") pod \"379c7f20-d197-4044-9afd-b3dec7ec86a6\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") "
Sep 10 15:04:28.183760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.183700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-model-cache\") pod \"379c7f20-d197-4044-9afd-b3dec7ec86a6\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") "
Sep 10 15:04:28.183760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.183729    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379c7f20-d197-4044-9afd-b3dec7ec86a6-tls-certs\") pod \"379c7f20-d197-4044-9afd-b3dec7ec86a6\" (UID: \"379c7f20-d197-4044-9afd-b3dec7ec86a6\") "
Sep 10 15:04:28.184043 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.183901    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "379c7f20-d197-4044-9afd-b3dec7ec86a6" (UID: "379c7f20-d197-4044-9afd-b3dec7ec86a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:28.184043 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.184008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "379c7f20-d197-4044-9afd-b3dec7ec86a6" (UID: "379c7f20-d197-4044-9afd-b3dec7ec86a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:28.184043 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.184027    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-home" (OuterVolumeSpecName: "home") pod "379c7f20-d197-4044-9afd-b3dec7ec86a6" (UID: "379c7f20-d197-4044-9afd-b3dec7ec86a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:28.184181 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.184068    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:28.185743 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.185722    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-dshm" (OuterVolumeSpecName: "dshm") pod "379c7f20-d197-4044-9afd-b3dec7ec86a6" (UID: "379c7f20-d197-4044-9afd-b3dec7ec86a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:28.185857 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.185838    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379c7f20-d197-4044-9afd-b3dec7ec86a6-kube-api-access-w4vbn" (OuterVolumeSpecName: "kube-api-access-w4vbn") pod "379c7f20-d197-4044-9afd-b3dec7ec86a6" (UID: "379c7f20-d197-4044-9afd-b3dec7ec86a6"). InnerVolumeSpecName "kube-api-access-w4vbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:28.185899 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.185882    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379c7f20-d197-4044-9afd-b3dec7ec86a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "379c7f20-d197-4044-9afd-b3dec7ec86a6" (UID: "379c7f20-d197-4044-9afd-b3dec7ec86a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:28.209295 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.209269    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:28.209407 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.209307    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:28.221888 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.221861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:28.284749 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.284668    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4vbn\" (UniqueName: \"kubernetes.io/projected/379c7f20-d197-4044-9afd-b3dec7ec86a6-kube-api-access-w4vbn\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:28.284749 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.284698    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:28.284749 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.284708    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:28.284749 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.284716    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/379c7f20-d197-4044-9afd-b3dec7ec86a6-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:28.284749 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.284725    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/379c7f20-d197-4044-9afd-b3dec7ec86a6-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:04:28.897595 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.897564    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"
Sep 10 15:04:28.898051 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.897607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq" event={"ID":"379c7f20-d197-4044-9afd-b3dec7ec86a6","Type":"ContainerDied","Data":"fe5316bfe1101cb2bc7737ef03c87ba035d72f79e67366acd30b44053091f85f"}
Sep 10 15:04:28.898051 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.897678    2565 scope.go:117] "RemoveContainer" containerID="b4292333e5d721c9d6519f00daa2d73a517ae9ab90cbdbabf32825450252f526"
Sep 10 15:04:28.910666 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.910629    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:04:28.920495 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.920466    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"]
Sep 10 15:04:28.927370 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:28.927345    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-c1b5b19d-kserve-66994b9f86-wjblq"]
Sep 10 15:04:29.977217 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:29.977185    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="379c7f20-d197-4044-9afd-b3dec7ec86a6" path="/var/lib/kubelet/pods/379c7f20-d197-4044-9afd-b3dec7ec86a6/volumes"
Sep 10 15:04:42.969087 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:42.969052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:04:59.673541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.673495    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"]
Sep 10 15:04:59.674091 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.674068    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379c7f20-d197-4044-9afd-b3dec7ec86a6" containerName="main"
Sep 10 15:04:59.674140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.674096    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="379c7f20-d197-4044-9afd-b3dec7ec86a6" containerName="main"
Sep 10 15:04:59.674140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.674133    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f827e6-6126-494e-b162-7b513bfab77e" containerName="main"
Sep 10 15:04:59.674236 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.674142    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f827e6-6126-494e-b162-7b513bfab77e" containerName="main"
Sep 10 15:04:59.674298 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.674245    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="379c7f20-d197-4044-9afd-b3dec7ec86a6" containerName="main"
Sep 10 15:04:59.674298 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.674257    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f827e6-6126-494e-b162-7b513bfab77e" containerName="main"
Sep 10 15:04:59.677668 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.677622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.680261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.680238    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-78f25d\"/\"openshift-service-ca.crt\""
Sep 10 15:04:59.680484 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.680466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-78f25d\"/\"kube-root-ca.crt\""
Sep 10 15:04:59.681061 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.681046    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-78f25d\"/\"default-dockercfg-z2p4q\""
Sep 10 15:04:59.682760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.682742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-78f25d\"/\"explicit-model-78f25d-kserve-self-signed-certs\""
Sep 10 15:04:59.688165 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.688141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"]
Sep 10 15:04:59.786279 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.786237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-tmp-dir\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.786468 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.786293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-dshm\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.786468 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.786354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-model-cache\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.786468 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.786431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-home\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.786606 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.786489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf241a-9cf6-4ebd-bf92-d598863ad231-tls-certs\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.786606 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.786509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsljr\" (UniqueName: \"kubernetes.io/projected/59bf241a-9cf6-4ebd-bf92-d598863ad231-kube-api-access-rsljr\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.887153 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-home\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.887337 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf241a-9cf6-4ebd-bf92-d598863ad231-tls-certs\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.887337 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsljr\" (UniqueName: \"kubernetes.io/projected/59bf241a-9cf6-4ebd-bf92-d598863ad231-kube-api-access-rsljr\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.887337 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-tmp-dir\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.887337 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-dshm\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.887337 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-model-cache\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.888240 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-home\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.888371 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.888251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-tmp-dir\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.888371 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.887728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-model-cache\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.895310 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.895279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-dshm\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.895945 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.895925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf241a-9cf6-4ebd-bf92-d598863ad231-tls-certs\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.903740 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.903718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsljr\" (UniqueName: \"kubernetes.io/projected/59bf241a-9cf6-4ebd-bf92-d598863ad231-kube-api-access-rsljr\") pod \"explicit-model-78f25d-kserve-5bdffb7489-29bkp\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") " pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:04:59.988026 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:04:59.987938    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:05:00.260240 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:00.260212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"]
Sep 10 15:05:00.262209 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:05:00.262188    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59bf241a_9cf6_4ebd_bf92_d598863ad231.slice/crio-745e7f4fd5f6370ea9832755ae59abbebc9dcadc965e1bcaaf8acdea238fd29d WatchSource:0}: Error finding container 745e7f4fd5f6370ea9832755ae59abbebc9dcadc965e1bcaaf8acdea238fd29d: Status 404 returned error can't find the container with id 745e7f4fd5f6370ea9832755ae59abbebc9dcadc965e1bcaaf8acdea238fd29d
Sep 10 15:05:00.663053 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:00.663017    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:05:01.069101 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:01.069009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp" event={"ID":"59bf241a-9cf6-4ebd-bf92-d598863ad231","Type":"ContainerStarted","Data":"e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada"}
Sep 10 15:05:01.069101 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:01.069048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp" event={"ID":"59bf241a-9cf6-4ebd-bf92-d598863ad231","Type":"ContainerStarted","Data":"745e7f4fd5f6370ea9832755ae59abbebc9dcadc965e1bcaaf8acdea238fd29d"}
Sep 10 15:05:01.088295 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:01.088243    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp" podStartSLOduration=2.088224183 podStartE2EDuration="2.088224183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:01.087232 +0000 UTC m=+1035.656771258" watchObservedRunningTime="2026-09-10 15:05:01.088224183 +0000 UTC m=+1035.657763443"
Sep 10 15:05:07.587361 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:07.587326    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"]
Sep 10 15:05:09.988919 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:09.988890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:05:16.587215 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:16.587175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:05:22.607247 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.607190    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp" podUID="59bf241a-9cf6-4ebd-bf92-d598863ad231" containerName="main" containerID="cri-o://e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada" gracePeriod=45
Sep 10 15:05:22.853978 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.853953    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:05:22.912243 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912161    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-model-cache\") pod \"59bf241a-9cf6-4ebd-bf92-d598863ad231\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") "
Sep 10 15:05:22.912243 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912196    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-dshm\") pod \"59bf241a-9cf6-4ebd-bf92-d598863ad231\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") "
Sep 10 15:05:22.912485 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912262    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-tmp-dir\") pod \"59bf241a-9cf6-4ebd-bf92-d598863ad231\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") "
Sep 10 15:05:22.912485 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsljr\" (UniqueName: \"kubernetes.io/projected/59bf241a-9cf6-4ebd-bf92-d598863ad231-kube-api-access-rsljr\") pod \"59bf241a-9cf6-4ebd-bf92-d598863ad231\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") "
Sep 10 15:05:22.912485 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912333    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf241a-9cf6-4ebd-bf92-d598863ad231-tls-certs\") pod \"59bf241a-9cf6-4ebd-bf92-d598863ad231\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") "
Sep 10 15:05:22.912485 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912377    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-home\") pod \"59bf241a-9cf6-4ebd-bf92-d598863ad231\" (UID: \"59bf241a-9cf6-4ebd-bf92-d598863ad231\") "
Sep 10 15:05:22.912741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-model-cache" (OuterVolumeSpecName: "model-cache") pod "59bf241a-9cf6-4ebd-bf92-d598863ad231" (UID: "59bf241a-9cf6-4ebd-bf92-d598863ad231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:22.912741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912509    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59bf241a-9cf6-4ebd-bf92-d598863ad231" (UID: "59bf241a-9cf6-4ebd-bf92-d598863ad231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:22.912741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912689    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:22.912741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912702    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:22.912741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.912724    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-home" (OuterVolumeSpecName: "home") pod "59bf241a-9cf6-4ebd-bf92-d598863ad231" (UID: "59bf241a-9cf6-4ebd-bf92-d598863ad231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:22.914563 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.914540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59bf241a-9cf6-4ebd-bf92-d598863ad231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59bf241a-9cf6-4ebd-bf92-d598863ad231" (UID: "59bf241a-9cf6-4ebd-bf92-d598863ad231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:05:22.914563 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.914549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bf241a-9cf6-4ebd-bf92-d598863ad231-kube-api-access-rsljr" (OuterVolumeSpecName: "kube-api-access-rsljr") pod "59bf241a-9cf6-4ebd-bf92-d598863ad231" (UID: "59bf241a-9cf6-4ebd-bf92-d598863ad231"). InnerVolumeSpecName "kube-api-access-rsljr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:22.914710 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:22.914551    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-dshm" (OuterVolumeSpecName: "dshm") pod "59bf241a-9cf6-4ebd-bf92-d598863ad231" (UID: "59bf241a-9cf6-4ebd-bf92-d598863ad231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:23.014087 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.014061    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bf241a-9cf6-4ebd-bf92-d598863ad231-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:23.014087 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.014083    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:23.014087 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.014091    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bf241a-9cf6-4ebd-bf92-d598863ad231-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:23.014282 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.014102    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsljr\" (UniqueName: \"kubernetes.io/projected/59bf241a-9cf6-4ebd-bf92-d598863ad231-kube-api-access-rsljr\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:23.185109 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.185027    2565 generic.go:356] "Generic (PLEG): container finished" podID="59bf241a-9cf6-4ebd-bf92-d598863ad231" containerID="e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada" exitCode=0
Sep 10 15:05:23.185109 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.185071    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"
Sep 10 15:05:23.185310 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.185106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp" event={"ID":"59bf241a-9cf6-4ebd-bf92-d598863ad231","Type":"ContainerDied","Data":"e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada"}
Sep 10 15:05:23.185310 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.185145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp" event={"ID":"59bf241a-9cf6-4ebd-bf92-d598863ad231","Type":"ContainerDied","Data":"745e7f4fd5f6370ea9832755ae59abbebc9dcadc965e1bcaaf8acdea238fd29d"}
Sep 10 15:05:23.185310 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.185162    2565 scope.go:117] "RemoveContainer" containerID="e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada"
Sep 10 15:05:23.194362 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.194344    2565 scope.go:117] "RemoveContainer" containerID="e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada"
Sep 10 15:05:23.194605 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:05:23.194587    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada\": container with ID starting with e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada not found: ID does not exist" containerID="e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada"
Sep 10 15:05:23.194670 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.194612    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada"} err="failed to get container status \"e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada\": rpc error: code = NotFound desc = could not find container \"e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada\": container with ID starting with e66eb0f761d8eee6793b8b4a67328957e9364d721c39b534fe2fa507b68c8ada not found: ID does not exist"
Sep 10 15:05:23.207788 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.207766    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"]
Sep 10 15:05:23.218209 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.218188    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-78f25d/explicit-model-78f25d-kserve-5bdffb7489-29bkp"]
Sep 10 15:05:23.977104 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:23.977042    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59bf241a-9cf6-4ebd-bf92-d598863ad231" path="/var/lib/kubelet/pods/59bf241a-9cf6-4ebd-bf92-d598863ad231/volumes"
Sep 10 15:05:34.763014 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:34.762982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:05:41.479711 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.479621    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"]
Sep 10 15:05:41.480127 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.480017    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59bf241a-9cf6-4ebd-bf92-d598863ad231" containerName="main"
Sep 10 15:05:41.480127 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.480033    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="59bf241a-9cf6-4ebd-bf92-d598863ad231" containerName="main"
Sep 10 15:05:41.480127 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.480106    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="59bf241a-9cf6-4ebd-bf92-d598863ad231" containerName="main"
Sep 10 15:05:41.484361 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.484342    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.487379 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.487359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-328f99\"/\"kube-root-ca.crt\""
Sep 10 15:05:41.488219 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.488197    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-328f99\"/\"openshift-service-ca.crt\""
Sep 10 15:05:41.488219 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.488212    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-328f99\"/\"default-dockercfg-hvrlt\""
Sep 10 15:05:41.490741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.490518    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-328f99\"/\"noten-model-328f99-kserve-self-signed-certs\""
Sep 10 15:05:41.573619 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.573591    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"]
Sep 10 15:05:41.587766 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.587737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-model-cache\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.587912 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.587774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zg27\" (UniqueName: \"kubernetes.io/projected/58e991bb-2046-442a-95a9-d0f01bee7cbe-kube-api-access-2zg27\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.587912 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.587826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-tmp-dir\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.587912 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.587869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e991bb-2046-442a-95a9-d0f01bee7cbe-tls-certs\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.588052 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.587932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-home\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.588052 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.588036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-dshm\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.688538 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.688496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-dshm\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.688757 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.688597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-model-cache\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.688757 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.688625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zg27\" (UniqueName: \"kubernetes.io/projected/58e991bb-2046-442a-95a9-d0f01bee7cbe-kube-api-access-2zg27\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.688757 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.688673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-tmp-dir\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.688757 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.688698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e991bb-2046-442a-95a9-d0f01bee7cbe-tls-certs\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.688757 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.688732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-home\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.689060 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.689032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-model-cache\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.689129 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.689090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-home\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.689192 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.689131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-tmp-dir\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.690930 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.690905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-dshm\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.691215 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.691198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e991bb-2046-442a-95a9-d0f01bee7cbe-tls-certs\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.697129 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.697103    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zg27\" (UniqueName: \"kubernetes.io/projected/58e991bb-2046-442a-95a9-d0f01bee7cbe-kube-api-access-2zg27\") pod \"noten-model-328f99-kserve-9d6955f9-kkwgk\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") " pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:41.792690 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:41.792559    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:42.072596 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:42.072570    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"]
Sep 10 15:05:42.074002 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:05:42.073973    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e991bb_2046_442a_95a9_d0f01bee7cbe.slice/crio-1eb3629736f3e6ae4d2e54b40a80441d9245d7c3d6df3d023e6333d687a656bf WatchSource:0}: Error finding container 1eb3629736f3e6ae4d2e54b40a80441d9245d7c3d6df3d023e6333d687a656bf: Status 404 returned error can't find the container with id 1eb3629736f3e6ae4d2e54b40a80441d9245d7c3d6df3d023e6333d687a656bf
Sep 10 15:05:42.284576 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:42.284538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk" event={"ID":"58e991bb-2046-442a-95a9-d0f01bee7cbe","Type":"ContainerStarted","Data":"5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e"}
Sep 10 15:05:42.284576 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:42.284583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk" event={"ID":"58e991bb-2046-442a-95a9-d0f01bee7cbe","Type":"ContainerStarted","Data":"1eb3629736f3e6ae4d2e54b40a80441d9245d7c3d6df3d023e6333d687a656bf"}
Sep 10 15:05:42.306361 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:42.306295    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk" podStartSLOduration=1.306269654 podStartE2EDuration="1.306269654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:05:42.302191386 +0000 UTC m=+1076.871730644" watchObservedRunningTime="2026-09-10 15:05:42.306269654 +0000 UTC m=+1076.875808913"
Sep 10 15:05:43.643013 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:43.642981    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"]
Sep 10 15:05:46.560767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:46.560736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"]
Sep 10 15:05:47.736047 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:47.736012    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"]
Sep 10 15:05:51.066165 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:51.066127    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:05:51.793715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:51.793680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:05:58.662289 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.662233    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng" podUID="9eb53c36-c341-4fea-90df-eceb3e063e14" containerName="main" containerID="cri-o://a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374" gracePeriod=45
Sep 10 15:05:58.909853 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.909826    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:05:58.952469 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952381    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-home\") pod \"9eb53c36-c341-4fea-90df-eceb3e063e14\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") "
Sep 10 15:05:58.952469 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952446    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th2gh\" (UniqueName: \"kubernetes.io/projected/9eb53c36-c341-4fea-90df-eceb3e063e14-kube-api-access-th2gh\") pod \"9eb53c36-c341-4fea-90df-eceb3e063e14\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") "
Sep 10 15:05:58.952752 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952489    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-model-cache\") pod \"9eb53c36-c341-4fea-90df-eceb3e063e14\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") "
Sep 10 15:05:58.952752 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952515    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-dshm\") pod \"9eb53c36-c341-4fea-90df-eceb3e063e14\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") "
Sep 10 15:05:58.952752 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-tmp-dir\") pod \"9eb53c36-c341-4fea-90df-eceb3e063e14\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") "
Sep 10 15:05:58.952752 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb53c36-c341-4fea-90df-eceb3e063e14-tls-certs\") pod \"9eb53c36-c341-4fea-90df-eceb3e063e14\" (UID: \"9eb53c36-c341-4fea-90df-eceb3e063e14\") "
Sep 10 15:05:58.952752 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952704    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-home" (OuterVolumeSpecName: "home") pod "9eb53c36-c341-4fea-90df-eceb3e063e14" (UID: "9eb53c36-c341-4fea-90df-eceb3e063e14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:58.953028 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952847    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9eb53c36-c341-4fea-90df-eceb3e063e14" (UID: "9eb53c36-c341-4fea-90df-eceb3e063e14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:58.953028 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.952984    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:58.953028 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.953004    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:58.953188 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.953068    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-model-cache" (OuterVolumeSpecName: "model-cache") pod "9eb53c36-c341-4fea-90df-eceb3e063e14" (UID: "9eb53c36-c341-4fea-90df-eceb3e063e14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:58.954740 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.954713    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-dshm" (OuterVolumeSpecName: "dshm") pod "9eb53c36-c341-4fea-90df-eceb3e063e14" (UID: "9eb53c36-c341-4fea-90df-eceb3e063e14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:58.955007 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.954984    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb53c36-c341-4fea-90df-eceb3e063e14-kube-api-access-th2gh" (OuterVolumeSpecName: "kube-api-access-th2gh") pod "9eb53c36-c341-4fea-90df-eceb3e063e14" (UID: "9eb53c36-c341-4fea-90df-eceb3e063e14"). InnerVolumeSpecName "kube-api-access-th2gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:58.955007 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:58.954981    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb53c36-c341-4fea-90df-eceb3e063e14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9eb53c36-c341-4fea-90df-eceb3e063e14" (UID: "9eb53c36-c341-4fea-90df-eceb3e063e14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:05:59.053516 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.053480    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th2gh\" (UniqueName: \"kubernetes.io/projected/9eb53c36-c341-4fea-90df-eceb3e063e14-kube-api-access-th2gh\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:59.053516 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.053511    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:59.053516 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.053520    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb53c36-c341-4fea-90df-eceb3e063e14-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:59.053767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.053528    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb53c36-c341-4fea-90df-eceb3e063e14-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:05:59.383261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.383231    2565 generic.go:356] "Generic (PLEG): container finished" podID="9eb53c36-c341-4fea-90df-eceb3e063e14" containerID="a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374" exitCode=0
Sep 10 15:05:59.383410 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.383274    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"
Sep 10 15:05:59.383410 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.383302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng" event={"ID":"9eb53c36-c341-4fea-90df-eceb3e063e14","Type":"ContainerDied","Data":"a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374"}
Sep 10 15:05:59.383410 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.383334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng" event={"ID":"9eb53c36-c341-4fea-90df-eceb3e063e14","Type":"ContainerDied","Data":"e979e1f7f153e0b700c42fbd4b4c9e7b31f2fc4969c5e96b964c1cc27c3cb38a"}
Sep 10 15:05:59.383410 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.383353    2565 scope.go:117] "RemoveContainer" containerID="a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374"
Sep 10 15:05:59.392795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.392769    2565 scope.go:117] "RemoveContainer" containerID="a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374"
Sep 10 15:05:59.393072 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:05:59.393054    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374\": container with ID starting with a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374 not found: ID does not exist" containerID="a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374"
Sep 10 15:05:59.393138 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.393086    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374"} err="failed to get container status \"a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374\": rpc error: code = NotFound desc = could not find container \"a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374\": container with ID starting with a6362bfe4e9823938be70444f1fbf16e5da6490a66494c18a4538907da291374 not found: ID does not exist"
Sep 10 15:05:59.405490 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.405461    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"]
Sep 10 15:05:59.409550 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.409527    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e577f0/test-model-e577f0-kserve-757fff4dd5-w79ng"]
Sep 10 15:05:59.976538 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:05:59.976504    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb53c36-c341-4fea-90df-eceb3e063e14" path="/var/lib/kubelet/pods/9eb53c36-c341-4fea-90df-eceb3e063e14/volumes"
Sep 10 15:06:01.579956 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.579917    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk" podUID="58e991bb-2046-442a-95a9-d0f01bee7cbe" containerName="main" containerID="cri-o://5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e" gracePeriod=45
Sep 10 15:06:01.755379 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.755348    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:06:01.829588 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.829564    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:06:01.879526 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879491    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-dshm\") pod \"58e991bb-2046-442a-95a9-d0f01bee7cbe\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") "
Sep 10 15:06:01.879715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879555    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-tmp-dir\") pod \"58e991bb-2046-442a-95a9-d0f01bee7cbe\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") "
Sep 10 15:06:01.879715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-home\") pod \"58e991bb-2046-442a-95a9-d0f01bee7cbe\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") "
Sep 10 15:06:01.879715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-model-cache\") pod \"58e991bb-2046-442a-95a9-d0f01bee7cbe\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") "
Sep 10 15:06:01.879715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879636    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e991bb-2046-442a-95a9-d0f01bee7cbe-tls-certs\") pod \"58e991bb-2046-442a-95a9-d0f01bee7cbe\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") "
Sep 10 15:06:01.879715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zg27\" (UniqueName: \"kubernetes.io/projected/58e991bb-2046-442a-95a9-d0f01bee7cbe-kube-api-access-2zg27\") pod \"58e991bb-2046-442a-95a9-d0f01bee7cbe\" (UID: \"58e991bb-2046-442a-95a9-d0f01bee7cbe\") "
Sep 10 15:06:01.879999 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58e991bb-2046-442a-95a9-d0f01bee7cbe" (UID: "58e991bb-2046-442a-95a9-d0f01bee7cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:01.879999 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879899    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-home" (OuterVolumeSpecName: "home") pod "58e991bb-2046-442a-95a9-d0f01bee7cbe" (UID: "58e991bb-2046-442a-95a9-d0f01bee7cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:01.879999 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.879930    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "58e991bb-2046-442a-95a9-d0f01bee7cbe" (UID: "58e991bb-2046-442a-95a9-d0f01bee7cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:01.880153 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.880123    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:01.880153 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.880141    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:01.880258 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.880153    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:01.881898 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.881870    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e991bb-2046-442a-95a9-d0f01bee7cbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58e991bb-2046-442a-95a9-d0f01bee7cbe" (UID: "58e991bb-2046-442a-95a9-d0f01bee7cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:06:01.882115 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.882097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-dshm" (OuterVolumeSpecName: "dshm") pod "58e991bb-2046-442a-95a9-d0f01bee7cbe" (UID: "58e991bb-2046-442a-95a9-d0f01bee7cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:01.882165 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.882123    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e991bb-2046-442a-95a9-d0f01bee7cbe-kube-api-access-2zg27" (OuterVolumeSpecName: "kube-api-access-2zg27") pod "58e991bb-2046-442a-95a9-d0f01bee7cbe" (UID: "58e991bb-2046-442a-95a9-d0f01bee7cbe"). InnerVolumeSpecName "kube-api-access-2zg27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:06:01.980702 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.980672    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e991bb-2046-442a-95a9-d0f01bee7cbe-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:01.980702 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.980697    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zg27\" (UniqueName: \"kubernetes.io/projected/58e991bb-2046-442a-95a9-d0f01bee7cbe-kube-api-access-2zg27\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:01.980702 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:01.980707    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e991bb-2046-442a-95a9-d0f01bee7cbe-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:02.400302 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.400267    2565 generic.go:356] "Generic (PLEG): container finished" podID="58e991bb-2046-442a-95a9-d0f01bee7cbe" containerID="5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e" exitCode=0
Sep 10 15:06:02.400450 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.400313    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"
Sep 10 15:06:02.400450 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.400348    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk" event={"ID":"58e991bb-2046-442a-95a9-d0f01bee7cbe","Type":"ContainerDied","Data":"5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e"}
Sep 10 15:06:02.400450 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.400390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk" event={"ID":"58e991bb-2046-442a-95a9-d0f01bee7cbe","Type":"ContainerDied","Data":"1eb3629736f3e6ae4d2e54b40a80441d9245d7c3d6df3d023e6333d687a656bf"}
Sep 10 15:06:02.400450 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.400408    2565 scope.go:117] "RemoveContainer" containerID="5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e"
Sep 10 15:06:02.408660 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.408632    2565 scope.go:117] "RemoveContainer" containerID="5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e"
Sep 10 15:06:02.408933 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:06:02.408912    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e\": container with ID starting with 5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e not found: ID does not exist" containerID="5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e"
Sep 10 15:06:02.409000 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.408945    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e"} err="failed to get container status \"5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e\": rpc error: code = NotFound desc = could not find container \"5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e\": container with ID starting with 5f0db3888430c93328ba13c582c03ef9d124dabd950581a218748a95684bce6e not found: ID does not exist"
Sep 10 15:06:02.426756 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.426732    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"]
Sep 10 15:06:02.436685 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.436662    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-328f99/noten-model-328f99-kserve-9d6955f9-kkwgk"]
Sep 10 15:06:02.755357 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:02.755256    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55" podUID="c27e2f98-9e31-4f2c-a939-7a3fcd10a031" containerName="main" containerID="cri-o://232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7" gracePeriod=45
Sep 10 15:06:03.003162 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.003139    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:06:03.091428 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tmp-dir\") pod \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") "
Sep 10 15:06:03.091428 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091387    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcnfb\" (UniqueName: \"kubernetes.io/projected/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-kube-api-access-qcnfb\") pod \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") "
Sep 10 15:06:03.091428 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091420    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-model-cache\") pod \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") "
Sep 10 15:06:03.091751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091438    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-home\") pod \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") "
Sep 10 15:06:03.091751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091500    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-dshm\") pod \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") "
Sep 10 15:06:03.091751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091550    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tls-certs\") pod \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\" (UID: \"c27e2f98-9e31-4f2c-a939-7a3fcd10a031\") "
Sep 10 15:06:03.091751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091673    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c27e2f98-9e31-4f2c-a939-7a3fcd10a031" (UID: "c27e2f98-9e31-4f2c-a939-7a3fcd10a031"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:03.091964 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091757    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-model-cache" (OuterVolumeSpecName: "model-cache") pod "c27e2f98-9e31-4f2c-a939-7a3fcd10a031" (UID: "c27e2f98-9e31-4f2c-a939-7a3fcd10a031"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:03.091964 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.091773    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-home" (OuterVolumeSpecName: "home") pod "c27e2f98-9e31-4f2c-a939-7a3fcd10a031" (UID: "c27e2f98-9e31-4f2c-a939-7a3fcd10a031"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:03.092549 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.092098    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:03.092549 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.092122    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:03.092549 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.092138    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:03.094041 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.094016    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c27e2f98-9e31-4f2c-a939-7a3fcd10a031" (UID: "c27e2f98-9e31-4f2c-a939-7a3fcd10a031"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:06:03.094150 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.094016    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-kube-api-access-qcnfb" (OuterVolumeSpecName: "kube-api-access-qcnfb") pod "c27e2f98-9e31-4f2c-a939-7a3fcd10a031" (UID: "c27e2f98-9e31-4f2c-a939-7a3fcd10a031"). InnerVolumeSpecName "kube-api-access-qcnfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:06:03.094150 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.094021    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-dshm" (OuterVolumeSpecName: "dshm") pod "c27e2f98-9e31-4f2c-a939-7a3fcd10a031" (UID: "c27e2f98-9e31-4f2c-a939-7a3fcd10a031"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:03.193152 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.193120    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:03.193152 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.193147    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnfb\" (UniqueName: \"kubernetes.io/projected/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-kube-api-access-qcnfb\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:03.193152 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.193156    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e2f98-9e31-4f2c-a939-7a3fcd10a031-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:03.407268 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.407235    2565 generic.go:356] "Generic (PLEG): container finished" podID="c27e2f98-9e31-4f2c-a939-7a3fcd10a031" containerID="232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7" exitCode=0
Sep 10 15:06:03.407478 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.407290    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"
Sep 10 15:06:03.407478 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.407322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55" event={"ID":"c27e2f98-9e31-4f2c-a939-7a3fcd10a031","Type":"ContainerDied","Data":"232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7"}
Sep 10 15:06:03.407478 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.407362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55" event={"ID":"c27e2f98-9e31-4f2c-a939-7a3fcd10a031","Type":"ContainerDied","Data":"e89fced963db76146f4b19a8146effcd357f9c786c20c7debac70c7d3802a645"}
Sep 10 15:06:03.407478 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.407381    2565 scope.go:117] "RemoveContainer" containerID="232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7"
Sep 10 15:06:03.416291 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.416276    2565 scope.go:117] "RemoveContainer" containerID="232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7"
Sep 10 15:06:03.416553 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:06:03.416531    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7\": container with ID starting with 232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7 not found: ID does not exist" containerID="232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7"
Sep 10 15:06:03.416602 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.416561    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7"} err="failed to get container status \"232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7\": rpc error: code = NotFound desc = could not find container \"232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7\": container with ID starting with 232cf24dd084795c515958a02f1d153dc57dbb261cc220ff65995c86d34ab5b7 not found: ID does not exist"
Sep 10 15:06:03.428498 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.428477    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"]
Sep 10 15:06:03.431926 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.431906    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6356e4/test-model-6356e4-kserve-794dbb9c6c-d4f55"]
Sep 10 15:06:03.977717 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.977684    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e991bb-2046-442a-95a9-d0f01bee7cbe" path="/var/lib/kubelet/pods/58e991bb-2046-442a-95a9-d0f01bee7cbe/volumes"
Sep 10 15:06:03.978088 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:03.978080    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27e2f98-9e31-4f2c-a939-7a3fcd10a031" path="/var/lib/kubelet/pods/c27e2f98-9e31-4f2c-a939-7a3fcd10a031/volumes"
Sep 10 15:06:16.423083 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.423043    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"]
Sep 10 15:06:16.423686 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.423304    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" podUID="5a4207e5-ca54-4847-b370-54e4fb327c76" containerName="maas-api" containerID="cri-o://acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f" gracePeriod=30
Sep 10 15:06:16.665326 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.665297    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:06:16.820902 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.820820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls\") pod \"5a4207e5-ca54-4847-b370-54e4fb327c76\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") "
Sep 10 15:06:16.820902 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.820897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-metrics-certs\") pod \"5a4207e5-ca54-4847-b370-54e4fb327c76\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") "
Sep 10 15:06:16.821082 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.820989    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4207e5-ca54-4847-b370-54e4fb327c76-openshift-service-ca\") pod \"5a4207e5-ca54-4847-b370-54e4fb327c76\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") "
Sep 10 15:06:16.821082 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.821071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62kzj\" (UniqueName: \"kubernetes.io/projected/5a4207e5-ca54-4847-b370-54e4fb327c76-kube-api-access-62kzj\") pod \"5a4207e5-ca54-4847-b370-54e4fb327c76\" (UID: \"5a4207e5-ca54-4847-b370-54e4fb327c76\") "
Sep 10 15:06:16.821334 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.821304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4207e5-ca54-4847-b370-54e4fb327c76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a4207e5-ca54-4847-b370-54e4fb327c76" (UID: "5a4207e5-ca54-4847-b370-54e4fb327c76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:06:16.823055 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.823025    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a4207e5-ca54-4847-b370-54e4fb327c76" (UID: "5a4207e5-ca54-4847-b370-54e4fb327c76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:06:16.823151 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.823063    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a4207e5-ca54-4847-b370-54e4fb327c76" (UID: "5a4207e5-ca54-4847-b370-54e4fb327c76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:06:16.823151 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.823084    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4207e5-ca54-4847-b370-54e4fb327c76-kube-api-access-62kzj" (OuterVolumeSpecName: "kube-api-access-62kzj") pod "5a4207e5-ca54-4847-b370-54e4fb327c76" (UID: "5a4207e5-ca54-4847-b370-54e4fb327c76"). InnerVolumeSpecName "kube-api-access-62kzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:06:16.921981 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.921951    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:16.921981 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.921975    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4207e5-ca54-4847-b370-54e4fb327c76-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:16.921981 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.921984    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4207e5-ca54-4847-b370-54e4fb327c76-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:16.922201 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:16.921993    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62kzj\" (UniqueName: \"kubernetes.io/projected/5a4207e5-ca54-4847-b370-54e4fb327c76-kube-api-access-62kzj\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:06:17.484306 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.484263    2565 generic.go:356] "Generic (PLEG): container finished" podID="5a4207e5-ca54-4847-b370-54e4fb327c76" containerID="acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f" exitCode=0
Sep 10 15:06:17.484306 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.484311    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"
Sep 10 15:06:17.484776 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.484346    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" event={"ID":"5a4207e5-ca54-4847-b370-54e4fb327c76","Type":"ContainerDied","Data":"acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f"}
Sep 10 15:06:17.484776 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.484411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx" event={"ID":"5a4207e5-ca54-4847-b370-54e4fb327c76","Type":"ContainerDied","Data":"7e3468a4fc4d8e96b3fea552727cbb94c68ef96e0e2cdd44795a43dda0c163d5"}
Sep 10 15:06:17.484776 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.484435    2565 scope.go:117] "RemoveContainer" containerID="acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f"
Sep 10 15:06:17.493542 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.493524    2565 scope.go:117] "RemoveContainer" containerID="acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f"
Sep 10 15:06:17.493828 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:06:17.493807    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f\": container with ID starting with acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f not found: ID does not exist" containerID="acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f"
Sep 10 15:06:17.493907 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.493839    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f"} err="failed to get container status \"acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f\": rpc error: code = NotFound desc = could not find container \"acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f\": container with ID starting with acb18f6ac92add444daf8e2eed39205481d9790b087ba87566256dfb498cd65f not found: ID does not exist"
Sep 10 15:06:17.508247 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.508222    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"]
Sep 10 15:06:17.510451 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.510431    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6356e4-6c9885685b-2snbx"]
Sep 10 15:06:17.977095 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:17.977059    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4207e5-ca54-4847-b370-54e4fb327c76" path="/var/lib/kubelet/pods/5a4207e5-ca54-4847-b370-54e4fb327c76/volumes"
Sep 10 15:06:19.750059 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:19.750024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:06:28.312424 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.312389    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"]
Sep 10 15:06:28.313693 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.313668    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27e2f98-9e31-4f2c-a939-7a3fcd10a031" containerName="main"
Sep 10 15:06:28.313847 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.313835    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27e2f98-9e31-4f2c-a939-7a3fcd10a031" containerName="main"
Sep 10 15:06:28.314000 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.313986    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb53c36-c341-4fea-90df-eceb3e063e14" containerName="main"
Sep 10 15:06:28.314072 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314063    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb53c36-c341-4fea-90df-eceb3e063e14" containerName="main"
Sep 10 15:06:28.314138 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314129    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e991bb-2046-442a-95a9-d0f01bee7cbe" containerName="main"
Sep 10 15:06:28.314201 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314193    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e991bb-2046-442a-95a9-d0f01bee7cbe" containerName="main"
Sep 10 15:06:28.314292 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314282    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4207e5-ca54-4847-b370-54e4fb327c76" containerName="maas-api"
Sep 10 15:06:28.314356 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314348    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4207e5-ca54-4847-b370-54e4fb327c76" containerName="maas-api"
Sep 10 15:06:28.314615 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314601    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb53c36-c341-4fea-90df-eceb3e063e14" containerName="main"
Sep 10 15:06:28.314748 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314737    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4207e5-ca54-4847-b370-54e4fb327c76" containerName="maas-api"
Sep 10 15:06:28.314823 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314814    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e991bb-2046-442a-95a9-d0f01bee7cbe" containerName="main"
Sep 10 15:06:28.314900 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.314891    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27e2f98-9e31-4f2c-a939-7a3fcd10a031" containerName="main"
Sep 10 15:06:28.320302 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.320280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.324474 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.324450    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-decc52-serving-cert\""
Sep 10 15:06:28.324591 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.324453    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:06:28.324591 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.324453    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:06:28.325320 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.325304    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:06:28.336591 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.336566    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vh2ph\""
Sep 10 15:06:28.340444 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.340419    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"]
Sep 10 15:06:28.421695 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.421663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.421870 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.421713    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.421870 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.421733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d9bc67-33d3-4f57-8f13-511a17d43468-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.421870 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.421847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mrq\" (UniqueName: \"kubernetes.io/projected/95d9bc67-33d3-4f57-8f13-511a17d43468-kube-api-access-p6mrq\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.523113 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.523079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.523113 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.523117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d9bc67-33d3-4f57-8f13-511a17d43468-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.523321 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.523187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrq\" (UniqueName: \"kubernetes.io/projected/95d9bc67-33d3-4f57-8f13-511a17d43468-kube-api-access-p6mrq\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.523321 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.523208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.523321 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:06:28.523301    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-serving-cert: secret "maas-api-e2e-shared-a-w3-decc52-serving-cert" not found
Sep 10 15:06:28.523448 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:06:28.523367    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls podName:95d9bc67-33d3-4f57-8f13-511a17d43468 nodeName:}" failed. No retries permitted until 2026-09-10 15:06:29.023346593 +0000 UTC m=+1123.592885851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls") pod "maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h" (UID: "95d9bc67-33d3-4f57-8f13-511a17d43468") : secret "maas-api-e2e-shared-a-w3-decc52-serving-cert" not found
Sep 10 15:06:28.523927 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.523908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d9bc67-33d3-4f57-8f13-511a17d43468-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.525434 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.525415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:28.541466 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:28.541437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mrq\" (UniqueName: \"kubernetes.io/projected/95d9bc67-33d3-4f57-8f13-511a17d43468-kube-api-access-p6mrq\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:29.028066 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:29.028023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:29.030457 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:29.030429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:29.230738 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:29.230708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:29.525382 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:29.525356    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"]
Sep 10 15:06:29.526736 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:06:29.526710    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d9bc67_33d3_4f57_8f13_511a17d43468.slice/crio-d4571b723d2545297a2dde1c6b52d79fe2493e174a909bb983790e58d557819a WatchSource:0}: Error finding container d4571b723d2545297a2dde1c6b52d79fe2493e174a909bb983790e58d557819a: Status 404 returned error can't find the container with id d4571b723d2545297a2dde1c6b52d79fe2493e174a909bb983790e58d557819a
Sep 10 15:06:29.528397 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:29.528379    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:06:29.549180 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:29.549151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h" event={"ID":"95d9bc67-33d3-4f57-8f13-511a17d43468","Type":"ContainerStarted","Data":"d4571b723d2545297a2dde1c6b52d79fe2493e174a909bb983790e58d557819a"}
Sep 10 15:06:30.556270 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:30.556236    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h" event={"ID":"95d9bc67-33d3-4f57-8f13-511a17d43468","Type":"ContainerStarted","Data":"832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805"}
Sep 10 15:06:30.556721 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:30.556353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:30.576153 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:30.576101    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h" podStartSLOduration=2.026185347 podStartE2EDuration="2.576085949s" totalImagesPullingTime="549.900602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:30.574808422 +0000 UTC m=+1125.144347697" watchObservedRunningTime="2026-09-10 15:06:30.576085949 +0000 UTC m=+1125.145625200"
Sep 10 15:06:31.457881 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:31.457845    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:06:36.567179 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:36.567149    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:06:40.966052 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:06:40.966015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:07:04.076395 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:04.076359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:07:25.654111 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.654076    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"]
Sep 10 15:07:25.657616 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.657595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.660196 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.660176    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-1f985c-serving-cert\""
Sep 10 15:07:25.670787 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.670763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"]
Sep 10 15:07:25.742797 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.742773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.742893 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.742815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmlpp\" (UniqueName: \"kubernetes.io/projected/32a2d075-98b6-4155-a704-b75646075be7-kube-api-access-vmlpp\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.742893 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.742856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a2d075-98b6-4155-a704-b75646075be7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.742975 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.742909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.843770 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.843741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.843893 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.843811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.843893 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.843842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmlpp\" (UniqueName: \"kubernetes.io/projected/32a2d075-98b6-4155-a704-b75646075be7-kube-api-access-vmlpp\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.843893 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.843858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a2d075-98b6-4155-a704-b75646075be7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.844052 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:25.843943    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-serving-cert: secret "maas-api-e2e-shared-b-w2-1f985c-serving-cert" not found
Sep 10 15:07:25.844052 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:25.844008    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls podName:32a2d075-98b6-4155-a704-b75646075be7 nodeName:}" failed. No retries permitted until 2026-09-10 15:07:26.343988882 +0000 UTC m=+1180.913528124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls") pod "maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb" (UID: "32a2d075-98b6-4155-a704-b75646075be7") : secret "maas-api-e2e-shared-b-w2-1f985c-serving-cert" not found
Sep 10 15:07:25.844410 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.844391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a2d075-98b6-4155-a704-b75646075be7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.846231 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.846211    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:25.852262 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:25.852243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmlpp\" (UniqueName: \"kubernetes.io/projected/32a2d075-98b6-4155-a704-b75646075be7-kube-api-access-vmlpp\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:26.348572 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:26.348540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:26.351031 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:26.350997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:26.567596 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:26.567567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:26.843469 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:26.843442    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"]
Sep 10 15:07:26.845022 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:07:26.844996    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a2d075_98b6_4155_a704_b75646075be7.slice/crio-3cd8829269d8e69d00b895455be72d1b8f205d980246649f10876e58eb7daa4d WatchSource:0}: Error finding container 3cd8829269d8e69d00b895455be72d1b8f205d980246649f10876e58eb7daa4d: Status 404 returned error can't find the container with id 3cd8829269d8e69d00b895455be72d1b8f205d980246649f10876e58eb7daa4d
Sep 10 15:07:26.861488 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:26.861459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb" event={"ID":"32a2d075-98b6-4155-a704-b75646075be7","Type":"ContainerStarted","Data":"3cd8829269d8e69d00b895455be72d1b8f205d980246649f10876e58eb7daa4d"}
Sep 10 15:07:27.868366 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:27.868331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb" event={"ID":"32a2d075-98b6-4155-a704-b75646075be7","Type":"ContainerStarted","Data":"c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e"}
Sep 10 15:07:27.868785 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:27.868465    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:27.886112 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:27.886063    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb" podStartSLOduration=2.5590444789999998 podStartE2EDuration="2.886049351s" totalImagesPullingTime="327.004872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:27.885030429 +0000 UTC m=+1182.454569687" watchObservedRunningTime="2026-09-10 15:07:27.886049351 +0000 UTC m=+1182.455588608"
Sep 10 15:07:30.563339 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:30.563302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:07:31.539583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.539554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"]
Sep 10 15:07:31.543253 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.543235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.545687 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.545663    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-decc52\"/\"kube-root-ca.crt\""
Sep 10 15:07:31.546601 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.546575    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-decc52\"/\"openshift-service-ca.crt\""
Sep 10 15:07:31.546601 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.546598    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-decc52\"/\"default-dockercfg-n7b47\""
Sep 10 15:07:31.546780 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.546575    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-decc52\"/\"auth-test-model-decc52-kserve-self-signed-certs\""
Sep 10 15:07:31.556877 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.556853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"]
Sep 10 15:07:31.596216 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.596193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-tmp-dir\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.596541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.596223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-model-cache\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.596541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.596247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-dshm\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.596541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.596297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thp4n\" (UniqueName: \"kubernetes.io/projected/2292c02e-7e1c-458a-bc36-a92dae54ce08-kube-api-access-thp4n\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.596541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.596367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-home\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.596541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.596445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2292c02e-7e1c-458a-bc36-a92dae54ce08-tls-certs\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.697775 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.697747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2292c02e-7e1c-458a-bc36-a92dae54ce08-tls-certs\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.697904 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.697790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-tmp-dir\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.697904 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.697814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-model-cache\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.697904 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.697843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-dshm\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.697904 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.697863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thp4n\" (UniqueName: \"kubernetes.io/projected/2292c02e-7e1c-458a-bc36-a92dae54ce08-kube-api-access-thp4n\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.698188 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.697914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-home\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.698245 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.698207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-tmp-dir\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.698285 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.698273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-model-cache\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.698324 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.698303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-home\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.700048 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.700030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-dshm\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.700422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.700401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2292c02e-7e1c-458a-bc36-a92dae54ce08-tls-certs\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.706537 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.706518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thp4n\" (UniqueName: \"kubernetes.io/projected/2292c02e-7e1c-458a-bc36-a92dae54ce08-kube-api-access-thp4n\") pod \"auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") " pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:31.851777 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:31.851750    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:32.128992 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:32.128958    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"]
Sep 10 15:07:32.130228 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:07:32.130204    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2292c02e_7e1c_458a_bc36_a92dae54ce08.slice/crio-e29abf05312a6134c336651b984f039cea319df474a6da42911147755f2bab08 WatchSource:0}: Error finding container e29abf05312a6134c336651b984f039cea319df474a6da42911147755f2bab08: Status 404 returned error can't find the container with id e29abf05312a6134c336651b984f039cea319df474a6da42911147755f2bab08
Sep 10 15:07:32.900355 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:32.900308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f" event={"ID":"2292c02e-7e1c-458a-bc36-a92dae54ce08","Type":"ContainerStarted","Data":"d4b49697f24dca816e813aa1759c9e894f2817ab74c7460efeb95b7e3f30892b"}
Sep 10 15:07:32.900772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:32.900367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f" event={"ID":"2292c02e-7e1c-458a-bc36-a92dae54ce08","Type":"ContainerStarted","Data":"e29abf05312a6134c336651b984f039cea319df474a6da42911147755f2bab08"}
Sep 10 15:07:32.919216 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:32.919170    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f" podStartSLOduration=1.9191550560000001 podStartE2EDuration="1.919155056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:32.918292131 +0000 UTC m=+1187.487831398" watchObservedRunningTime="2026-09-10 15:07:32.919155056 +0000 UTC m=+1187.488694321"
Sep 10 15:07:33.879074 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:33.879043    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:07:41.851954 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:41.851914    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:41.853131 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:41.852013    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:41.855376 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:41.855346    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:07:41.864678 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:41.864650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:41.958581 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:41.958554    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:07:50.225418 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.225381    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7543-25fdf6"]
Sep 10 15:07:50.229245 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.229216    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-25fdf6"
Sep 10 15:07:50.232673 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.232652    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrlvr\""
Sep 10 15:07:50.238103 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.238080    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-25fdf6"]
Sep 10 15:07:50.277326 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.277291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjkl\" (UniqueName: \"kubernetes.io/projected/6007b78a-7979-4d67-8999-67f8356fafe1-kube-api-access-lpjkl\") pod \"test-curl-7543-25fdf6\" (UID: \"6007b78a-7979-4d67-8999-67f8356fafe1\") " pod="opendatahub/test-curl-7543-25fdf6"
Sep 10 15:07:50.378823 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.378789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjkl\" (UniqueName: \"kubernetes.io/projected/6007b78a-7979-4d67-8999-67f8356fafe1-kube-api-access-lpjkl\") pod \"test-curl-7543-25fdf6\" (UID: \"6007b78a-7979-4d67-8999-67f8356fafe1\") " pod="opendatahub/test-curl-7543-25fdf6"
Sep 10 15:07:50.388448 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.388420    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjkl\" (UniqueName: \"kubernetes.io/projected/6007b78a-7979-4d67-8999-67f8356fafe1-kube-api-access-lpjkl\") pod \"test-curl-7543-25fdf6\" (UID: \"6007b78a-7979-4d67-8999-67f8356fafe1\") " pod="opendatahub/test-curl-7543-25fdf6"
Sep 10 15:07:50.538731 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.538618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-25fdf6"
Sep 10 15:07:50.843006 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.836008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-25fdf6"]
Sep 10 15:07:50.843006 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:07:50.836408    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6007b78a_7979_4d67_8999_67f8356fafe1.slice/crio-35d17e5cec8a23d9899b3bacbc8cc2c6af918682247f9352f27f208078640952 WatchSource:0}: Error finding container 35d17e5cec8a23d9899b3bacbc8cc2c6af918682247f9352f27f208078640952: Status 404 returned error can't find the container with id 35d17e5cec8a23d9899b3bacbc8cc2c6af918682247f9352f27f208078640952
Sep 10 15:07:50.995278 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.995241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-25fdf6" event={"ID":"6007b78a-7979-4d67-8999-67f8356fafe1","Type":"ContainerStarted","Data":"36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b"}
Sep 10 15:07:50.995278 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:50.995290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-25fdf6" event={"ID":"6007b78a-7979-4d67-8999-67f8356fafe1","Type":"ContainerStarted","Data":"35d17e5cec8a23d9899b3bacbc8cc2c6af918682247f9352f27f208078640952"}
Sep 10 15:07:51.011286 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.011234    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7543-25fdf6" podStartSLOduration=1.011219032 podStartE2EDuration="1.011219032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:51.009817136 +0000 UTC m=+1205.579356394" watchObservedRunningTime="2026-09-10 15:07:51.011219032 +0000 UTC m=+1205.580758286"
Sep 10 15:07:51.618914 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.618876    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7543-25fdf6"]
Sep 10 15:07:51.622925 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.622899    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7543-25fdf6"]
Sep 10 15:07:51.736764 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.736724    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7543-f9f188"]
Sep 10 15:07:51.737158 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.737140    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6007b78a-7979-4d67-8999-67f8356fafe1" containerName="test-curl-7543-25fdf6"
Sep 10 15:07:51.737258 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.737163    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6007b78a-7979-4d67-8999-67f8356fafe1" containerName="test-curl-7543-25fdf6"
Sep 10 15:07:51.737313 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.737269    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6007b78a-7979-4d67-8999-67f8356fafe1" containerName="test-curl-7543-25fdf6"
Sep 10 15:07:51.740330 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.740308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f9f188"
Sep 10 15:07:51.745194 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.745169    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-f9f188"]
Sep 10 15:07:51.789629 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.789573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cm7q\" (UniqueName: \"kubernetes.io/projected/0f39d45d-1382-45c2-bd44-802cb9e62230-kube-api-access-4cm7q\") pod \"test-curl-7543-f9f188\" (UID: \"0f39d45d-1382-45c2-bd44-802cb9e62230\") " pod="opendatahub/test-curl-7543-f9f188"
Sep 10 15:07:51.890349 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.890263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cm7q\" (UniqueName: \"kubernetes.io/projected/0f39d45d-1382-45c2-bd44-802cb9e62230-kube-api-access-4cm7q\") pod \"test-curl-7543-f9f188\" (UID: \"0f39d45d-1382-45c2-bd44-802cb9e62230\") " pod="opendatahub/test-curl-7543-f9f188"
Sep 10 15:07:51.900239 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:51.900215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cm7q\" (UniqueName: \"kubernetes.io/projected/0f39d45d-1382-45c2-bd44-802cb9e62230-kube-api-access-4cm7q\") pod \"test-curl-7543-f9f188\" (UID: \"0f39d45d-1382-45c2-bd44-802cb9e62230\") " pod="opendatahub/test-curl-7543-f9f188"
Sep 10 15:07:52.000501 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:52.000473    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7543-25fdf6" container="test-curl-7543-25fdf6"
Sep 10 15:07:52.051102 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:52.051078    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f9f188"
Sep 10 15:07:52.323372 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:52.323346    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-f9f188"]
Sep 10 15:07:52.325283 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:07:52.325253    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f39d45d_1382_45c2_bd44_802cb9e62230.slice/crio-dc93f0b45b6f2dd7afdaa1d62cf94178eb1df2b5272d041ca85d9b044b6c84f2 WatchSource:0}: Error finding container dc93f0b45b6f2dd7afdaa1d62cf94178eb1df2b5272d041ca85d9b044b6c84f2: Status 404 returned error can't find the container with id dc93f0b45b6f2dd7afdaa1d62cf94178eb1df2b5272d041ca85d9b044b6c84f2
Sep 10 15:07:53.007595 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.007546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-f9f188" event={"ID":"0f39d45d-1382-45c2-bd44-802cb9e62230","Type":"ContainerStarted","Data":"6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c"}
Sep 10 15:07:53.007595 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.007602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-f9f188" event={"ID":"0f39d45d-1382-45c2-bd44-802cb9e62230","Type":"ContainerStarted","Data":"dc93f0b45b6f2dd7afdaa1d62cf94178eb1df2b5272d041ca85d9b044b6c84f2"}
Sep 10 15:07:53.008117 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.007665    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7543-25fdf6" podUID="6007b78a-7979-4d67-8999-67f8356fafe1" containerName="test-curl-7543-25fdf6" containerID="cri-o://36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b" gracePeriod=2
Sep 10 15:07:53.022309 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.022264    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7543-f9f188" podStartSLOduration=2.02224688 podStartE2EDuration="2.02224688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:53.021359733 +0000 UTC m=+1207.590898993" watchObservedRunningTime="2026-09-10 15:07:53.02224688 +0000 UTC m=+1207.591786138"
Sep 10 15:07:53.649932 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.649888    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7543-f9f188"]
Sep 10 15:07:53.654219 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.654189    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7543-f9f188"]
Sep 10 15:07:53.768562 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.768466    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7543-f95cc6"]
Sep 10 15:07:53.768975 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.768954    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f39d45d-1382-45c2-bd44-802cb9e62230" containerName="test-curl-7543-f9f188"
Sep 10 15:07:53.769067 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.768977    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f39d45d-1382-45c2-bd44-802cb9e62230" containerName="test-curl-7543-f9f188"
Sep 10 15:07:53.769131 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.769065    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f39d45d-1382-45c2-bd44-802cb9e62230" containerName="test-curl-7543-f9f188"
Sep 10 15:07:53.772231 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.772210    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f95cc6"
Sep 10 15:07:53.776274 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.776248    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-f95cc6"]
Sep 10 15:07:53.806254 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.806229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfsz4\" (UniqueName: \"kubernetes.io/projected/099ff028-9f44-4958-87ab-9267ee4049ee-kube-api-access-cfsz4\") pod \"test-curl-7543-f95cc6\" (UID: \"099ff028-9f44-4958-87ab-9267ee4049ee\") " pod="opendatahub/test-curl-7543-f95cc6"
Sep 10 15:07:53.907489 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.907449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsz4\" (UniqueName: \"kubernetes.io/projected/099ff028-9f44-4958-87ab-9267ee4049ee-kube-api-access-cfsz4\") pod \"test-curl-7543-f95cc6\" (UID: \"099ff028-9f44-4958-87ab-9267ee4049ee\") " pod="opendatahub/test-curl-7543-f95cc6"
Sep 10 15:07:53.919458 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:53.919426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfsz4\" (UniqueName: \"kubernetes.io/projected/099ff028-9f44-4958-87ab-9267ee4049ee-kube-api-access-cfsz4\") pod \"test-curl-7543-f95cc6\" (UID: \"099ff028-9f44-4958-87ab-9267ee4049ee\") " pod="opendatahub/test-curl-7543-f95cc6"
Sep 10 15:07:54.012724 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:54.012698    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7543-f9f188" container="test-curl-7543-f9f188"
Sep 10 15:07:54.083231 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:54.083126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f95cc6"
Sep 10 15:07:54.356943 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:54.356912    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-f95cc6"]
Sep 10 15:07:54.358308 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:07:54.358277    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod099ff028_9f44_4958_87ab_9267ee4049ee.slice/crio-634b2cc72206f2502abc45ca7cb05872b3478671f05ede5fccfa67a7abbb5724 WatchSource:0}: Error finding container 634b2cc72206f2502abc45ca7cb05872b3478671f05ede5fccfa67a7abbb5724: Status 404 returned error can't find the container with id 634b2cc72206f2502abc45ca7cb05872b3478671f05ede5fccfa67a7abbb5724
Sep 10 15:07:55.019709 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.019674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-f95cc6" event={"ID":"099ff028-9f44-4958-87ab-9267ee4049ee","Type":"ContainerStarted","Data":"5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12"}
Sep 10 15:07:55.020082 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.019724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-f95cc6" event={"ID":"099ff028-9f44-4958-87ab-9267ee4049ee","Type":"ContainerStarted","Data":"634b2cc72206f2502abc45ca7cb05872b3478671f05ede5fccfa67a7abbb5724"}
Sep 10 15:07:55.020082 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.019717    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7543-f9f188" podUID="0f39d45d-1382-45c2-bd44-802cb9e62230" containerName="test-curl-7543-f9f188" containerID="cri-o://6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c" gracePeriod=2
Sep 10 15:07:55.037895 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.037841    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7543-f95cc6" podStartSLOduration=2.037826069 podStartE2EDuration="2.037826069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:55.035313301 +0000 UTC m=+1209.604852560" watchObservedRunningTime="2026-09-10 15:07:55.037826069 +0000 UTC m=+1209.607365326"
Sep 10 15:07:55.180065 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.180035    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-25fdf6"
Sep 10 15:07:55.318360 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.318267    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpjkl\" (UniqueName: \"kubernetes.io/projected/6007b78a-7979-4d67-8999-67f8356fafe1-kube-api-access-lpjkl\") pod \"6007b78a-7979-4d67-8999-67f8356fafe1\" (UID: \"6007b78a-7979-4d67-8999-67f8356fafe1\") "
Sep 10 15:07:55.320274 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.320246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6007b78a-7979-4d67-8999-67f8356fafe1-kube-api-access-lpjkl" (OuterVolumeSpecName: "kube-api-access-lpjkl") pod "6007b78a-7979-4d67-8999-67f8356fafe1" (UID: "6007b78a-7979-4d67-8999-67f8356fafe1"). InnerVolumeSpecName "kube-api-access-lpjkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:07:55.418991 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.418969    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjkl\" (UniqueName: \"kubernetes.io/projected/6007b78a-7979-4d67-8999-67f8356fafe1-kube-api-access-lpjkl\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:07:55.627532 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.627480    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7543-f95cc6"]
Sep 10 15:07:55.631514 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.631481    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7543-f95cc6"]
Sep 10 15:07:55.757232 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.757198    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7543-e2a3b4"]
Sep 10 15:07:55.757581 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.757569    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099ff028-9f44-4958-87ab-9267ee4049ee" containerName="test-curl-7543-f95cc6"
Sep 10 15:07:55.757625 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.757583    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="099ff028-9f44-4958-87ab-9267ee4049ee" containerName="test-curl-7543-f95cc6"
Sep 10 15:07:55.757695 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.757681    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="099ff028-9f44-4958-87ab-9267ee4049ee" containerName="test-curl-7543-f95cc6"
Sep 10 15:07:55.760781 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.760765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-e2a3b4"
Sep 10 15:07:55.765721 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.765697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-e2a3b4"]
Sep 10 15:07:55.822571 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.822530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfllv\" (UniqueName: \"kubernetes.io/projected/4bad8aee-c18b-4056-956f-4422a0bd6731-kube-api-access-nfllv\") pod \"test-curl-7543-e2a3b4\" (UID: \"4bad8aee-c18b-4056-956f-4422a0bd6731\") " pod="opendatahub/test-curl-7543-e2a3b4"
Sep 10 15:07:55.923202 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.923117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfllv\" (UniqueName: \"kubernetes.io/projected/4bad8aee-c18b-4056-956f-4422a0bd6731-kube-api-access-nfllv\") pod \"test-curl-7543-e2a3b4\" (UID: \"4bad8aee-c18b-4056-956f-4422a0bd6731\") " pod="opendatahub/test-curl-7543-e2a3b4"
Sep 10 15:07:55.931687 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.931664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfllv\" (UniqueName: \"kubernetes.io/projected/4bad8aee-c18b-4056-956f-4422a0bd6731-kube-api-access-nfllv\") pod \"test-curl-7543-e2a3b4\" (UID: \"4bad8aee-c18b-4056-956f-4422a0bd6731\") " pod="opendatahub/test-curl-7543-e2a3b4"
Sep 10 15:07:55.977452 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:55.977424    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6007b78a-7979-4d67-8999-67f8356fafe1" path="/var/lib/kubelet/pods/6007b78a-7979-4d67-8999-67f8356fafe1/volumes"
Sep 10 15:07:56.026104 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:56.026078    2565 generic.go:356] "Generic (PLEG): container finished" podID="6007b78a-7979-4d67-8999-67f8356fafe1" containerID="36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b" exitCode=137
Sep 10 15:07:56.026429 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:56.026115    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-25fdf6"
Sep 10 15:07:56.026429 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:56.026170    2565 scope.go:117] "RemoveContainer" containerID="36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b"
Sep 10 15:07:56.026429 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:56.026375    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7543-f95cc6" container="test-curl-7543-f95cc6"
Sep 10 15:07:56.034944 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:56.034929    2565 scope.go:117] "RemoveContainer" containerID="36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b"
Sep 10 15:07:56.035190 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:56.035172    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b\": container with ID starting with 36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b not found: ID does not exist" containerID="36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b"
Sep 10 15:07:56.035246 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:56.035197    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b"} err="failed to get container status \"36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b\": rpc error: code = NotFound desc = could not find container \"36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b\": container with ID starting with 36e319d02ae08b40a5afca6a25cb75bfb46cac176be8690c6265864ab774149b not found: ID does not exist"
Sep 10 15:07:56.071288 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:56.071264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-e2a3b4"
Sep 10 15:07:56.344814 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:56.344786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-e2a3b4"]
Sep 10 15:07:56.346450 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:07:56.346419    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bad8aee_c18b_4056_956f_4422a0bd6731.slice/crio-202684f4681717ffba5540e0b520ceaae74f97327f9a55fc743cae31df6e3541 WatchSource:0}: Error finding container 202684f4681717ffba5540e0b520ceaae74f97327f9a55fc743cae31df6e3541: Status 404 returned error can't find the container with id 202684f4681717ffba5540e0b520ceaae74f97327f9a55fc743cae31df6e3541
Sep 10 15:07:57.034200 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.034167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-e2a3b4" event={"ID":"4bad8aee-c18b-4056-956f-4422a0bd6731","Type":"ContainerStarted","Data":"bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46"}
Sep 10 15:07:57.034528 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.034214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-e2a3b4" event={"ID":"4bad8aee-c18b-4056-956f-4422a0bd6731","Type":"ContainerStarted","Data":"202684f4681717ffba5540e0b520ceaae74f97327f9a55fc743cae31df6e3541"}
Sep 10 15:07:57.034528 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.034225    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7543-f95cc6" podUID="099ff028-9f44-4958-87ab-9267ee4049ee" containerName="test-curl-7543-f95cc6" containerID="cri-o://5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12" gracePeriod=2
Sep 10 15:07:57.049264 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.049220    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7543-e2a3b4" podStartSLOduration=2.049206949 podStartE2EDuration="2.049206949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:57.048320148 +0000 UTC m=+1211.617859407" watchObservedRunningTime="2026-09-10 15:07:57.049206949 +0000 UTC m=+1211.618746207"
Sep 10 15:07:57.204082 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.204059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f9f188"
Sep 10 15:07:57.234313 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.234289    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cm7q\" (UniqueName: \"kubernetes.io/projected/0f39d45d-1382-45c2-bd44-802cb9e62230-kube-api-access-4cm7q\") pod \"0f39d45d-1382-45c2-bd44-802cb9e62230\" (UID: \"0f39d45d-1382-45c2-bd44-802cb9e62230\") "
Sep 10 15:07:57.236167 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.236149    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f39d45d-1382-45c2-bd44-802cb9e62230-kube-api-access-4cm7q" (OuterVolumeSpecName: "kube-api-access-4cm7q") pod "0f39d45d-1382-45c2-bd44-802cb9e62230" (UID: "0f39d45d-1382-45c2-bd44-802cb9e62230"). InnerVolumeSpecName "kube-api-access-4cm7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:07:57.335472 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.335393    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cm7q\" (UniqueName: \"kubernetes.io/projected/0f39d45d-1382-45c2-bd44-802cb9e62230-kube-api-access-4cm7q\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:07:57.633501 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.633464    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7543-e2a3b4"]
Sep 10 15:07:57.638973 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.638947    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7543-e2a3b4"]
Sep 10 15:07:57.976478 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:57.976386    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f39d45d-1382-45c2-bd44-802cb9e62230" path="/var/lib/kubelet/pods/0f39d45d-1382-45c2-bd44-802cb9e62230/volumes"
Sep 10 15:07:58.040189 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:58.040151    2565 generic.go:356] "Generic (PLEG): container finished" podID="0f39d45d-1382-45c2-bd44-802cb9e62230" containerID="6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c" exitCode=137
Sep 10 15:07:58.040634 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:58.040192    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f9f188"
Sep 10 15:07:58.040634 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:58.040249    2565 scope.go:117] "RemoveContainer" containerID="6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c"
Sep 10 15:07:58.040634 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:58.040400    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7543-e2a3b4" container="test-curl-7543-e2a3b4"
Sep 10 15:07:58.049077 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:58.049059    2565 scope.go:117] "RemoveContainer" containerID="6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c"
Sep 10 15:07:58.049363 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:07:58.049344    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c\": container with ID starting with 6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c not found: ID does not exist" containerID="6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c"
Sep 10 15:07:58.049433 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:58.049371    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c"} err="failed to get container status \"6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c\": rpc error: code = NotFound desc = could not find container \"6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c\": container with ID starting with 6d4e28ae0c471a397313f72859119ebd81b6e1125a0dbe6b04b712e98444b81c not found: ID does not exist"
Sep 10 15:07:59.047013 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:59.046957    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7543-e2a3b4" podUID="4bad8aee-c18b-4056-956f-4422a0bd6731" containerName="test-curl-7543-e2a3b4" containerID="cri-o://bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46" gracePeriod=2
Sep 10 15:07:59.215873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:59.215849    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f95cc6"
Sep 10 15:07:59.253454 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:59.253427    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfsz4\" (UniqueName: \"kubernetes.io/projected/099ff028-9f44-4958-87ab-9267ee4049ee-kube-api-access-cfsz4\") pod \"099ff028-9f44-4958-87ab-9267ee4049ee\" (UID: \"099ff028-9f44-4958-87ab-9267ee4049ee\") "
Sep 10 15:07:59.255428 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:59.255405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099ff028-9f44-4958-87ab-9267ee4049ee-kube-api-access-cfsz4" (OuterVolumeSpecName: "kube-api-access-cfsz4") pod "099ff028-9f44-4958-87ab-9267ee4049ee" (UID: "099ff028-9f44-4958-87ab-9267ee4049ee"). InnerVolumeSpecName "kube-api-access-cfsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:07:59.355116 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:59.355088    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfsz4\" (UniqueName: \"kubernetes.io/projected/099ff028-9f44-4958-87ab-9267ee4049ee-kube-api-access-cfsz4\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:07:59.977121 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:07:59.977076    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099ff028-9f44-4958-87ab-9267ee4049ee" path="/var/lib/kubelet/pods/099ff028-9f44-4958-87ab-9267ee4049ee/volumes"
Sep 10 15:08:00.052897 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:00.052862    2565 generic.go:356] "Generic (PLEG): container finished" podID="099ff028-9f44-4958-87ab-9267ee4049ee" containerID="5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12" exitCode=137
Sep 10 15:08:00.053284 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:00.052907    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-f95cc6"
Sep 10 15:08:00.053284 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:00.052956    2565 scope.go:117] "RemoveContainer" containerID="5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12"
Sep 10 15:08:00.061480 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:00.061464    2565 scope.go:117] "RemoveContainer" containerID="5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12"
Sep 10 15:08:00.061767 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:08:00.061749    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12\": container with ID starting with 5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12 not found: ID does not exist" containerID="5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12"
Sep 10 15:08:00.061814 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:00.061776    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12"} err="failed to get container status \"5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12\": rpc error: code = NotFound desc = could not find container \"5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12\": container with ID starting with 5ec987fe5eee5030cadeb346f37eae1c6cb6d7c45f5b2c51a1cc6e80d502cd12 not found: ID does not exist"
Sep 10 15:08:01.241361 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.241333    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-e2a3b4"
Sep 10 15:08:01.273607 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.273576    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfllv\" (UniqueName: \"kubernetes.io/projected/4bad8aee-c18b-4056-956f-4422a0bd6731-kube-api-access-nfllv\") pod \"4bad8aee-c18b-4056-956f-4422a0bd6731\" (UID: \"4bad8aee-c18b-4056-956f-4422a0bd6731\") "
Sep 10 15:08:01.275670 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.275634    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bad8aee-c18b-4056-956f-4422a0bd6731-kube-api-access-nfllv" (OuterVolumeSpecName: "kube-api-access-nfllv") pod "4bad8aee-c18b-4056-956f-4422a0bd6731" (UID: "4bad8aee-c18b-4056-956f-4422a0bd6731"). InnerVolumeSpecName "kube-api-access-nfllv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:08:01.374473 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.374437    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfllv\" (UniqueName: \"kubernetes.io/projected/4bad8aee-c18b-4056-956f-4422a0bd6731-kube-api-access-nfllv\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:08:01.454277 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.454238    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"]
Sep 10 15:08:01.454628 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.454615    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bad8aee-c18b-4056-956f-4422a0bd6731" containerName="test-curl-7543-e2a3b4"
Sep 10 15:08:01.454698 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.454630    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bad8aee-c18b-4056-956f-4422a0bd6731" containerName="test-curl-7543-e2a3b4"
Sep 10 15:08:01.454738 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.454722    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bad8aee-c18b-4056-956f-4422a0bd6731" containerName="test-curl-7543-e2a3b4"
Sep 10 15:08:01.457800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.457784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.460350 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.460325    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"kube-root-ca.crt\""
Sep 10 15:08:01.461304 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.461282    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"openshift-service-ca.crt\""
Sep 10 15:08:01.461436 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.461289    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"auth-test-model-d5a2df-kserve-self-signed-certs\""
Sep 10 15:08:01.461436 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.461385    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"default-dockercfg-wfhrm\""
Sep 10 15:08:01.468408 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.468387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"]
Sep 10 15:08:01.474974 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.474951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndm2\" (UniqueName: \"kubernetes.io/projected/7d57277e-4fc7-405a-a983-381b0a9291fe-kube-api-access-tndm2\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.475127 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.474988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-tmp-dir\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.475127 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.475017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-model-cache\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.475127 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.475104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-dshm\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.475245 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.475157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d57277e-4fc7-405a-a983-381b0a9291fe-tls-certs\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.475245 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.475227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-home\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576063 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndm2\" (UniqueName: \"kubernetes.io/projected/7d57277e-4fc7-405a-a983-381b0a9291fe-kube-api-access-tndm2\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576063 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-tmp-dir\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-model-cache\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-dshm\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d57277e-4fc7-405a-a983-381b0a9291fe-tls-certs\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-home\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576582 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-model-cache\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576680 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-tmp-dir\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.576680 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.576611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-home\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.578345 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.578322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-dshm\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.578583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.578565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d57277e-4fc7-405a-a983-381b0a9291fe-tls-certs\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.585274 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.585251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndm2\" (UniqueName: \"kubernetes.io/projected/7d57277e-4fc7-405a-a983-381b0a9291fe-kube-api-access-tndm2\") pod \"auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.767106 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.767019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:01.977011 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:01.976973    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bad8aee-c18b-4056-956f-4422a0bd6731" path="/var/lib/kubelet/pods/4bad8aee-c18b-4056-956f-4422a0bd6731/volumes"
Sep 10 15:08:02.043290 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.043219    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"]
Sep 10 15:08:02.046429 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:08:02.046404    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d57277e_4fc7_405a_a983_381b0a9291fe.slice/crio-2ad101f3a179d42a0a0d865f177d3e46bf7f081d91736f43a2195bb2588be66f WatchSource:0}: Error finding container 2ad101f3a179d42a0a0d865f177d3e46bf7f081d91736f43a2195bb2588be66f: Status 404 returned error can't find the container with id 2ad101f3a179d42a0a0d865f177d3e46bf7f081d91736f43a2195bb2588be66f
Sep 10 15:08:02.066800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.066585    2565 generic.go:356] "Generic (PLEG): container finished" podID="4bad8aee-c18b-4056-956f-4422a0bd6731" containerID="bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46" exitCode=137
Sep 10 15:08:02.066800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.066708    2565 scope.go:117] "RemoveContainer" containerID="bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46"
Sep 10 15:08:02.066960 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.066928    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-e2a3b4"
Sep 10 15:08:02.068391 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.068356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh" event={"ID":"7d57277e-4fc7-405a-a983-381b0a9291fe","Type":"ContainerStarted","Data":"2ad101f3a179d42a0a0d865f177d3e46bf7f081d91736f43a2195bb2588be66f"}
Sep 10 15:08:02.081008 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.080992    2565 scope.go:117] "RemoveContainer" containerID="bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46"
Sep 10 15:08:02.081247 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:08:02.081225    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46\": container with ID starting with bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46 not found: ID does not exist" containerID="bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46"
Sep 10 15:08:02.081294 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.081259    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46"} err="failed to get container status \"bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46\": rpc error: code = NotFound desc = could not find container \"bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46\": container with ID starting with bf9c4852883593e07aa12b221e4c51b91965ae6ac2d050d44ee47414e973af46 not found: ID does not exist"
Sep 10 15:08:02.862483 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.862445    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"]
Sep 10 15:08:02.866173 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.866150    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.868343 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.868325    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5c801c28-serving-cert\""
Sep 10 15:08:02.879736 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.879709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"]
Sep 10 15:08:02.888581 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.888559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-metrics-certs\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.888745 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.888588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-openshift-service-ca\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.888745 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.888685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.888745 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.888702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77bw8\" (UniqueName: \"kubernetes.io/projected/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-kube-api-access-77bw8\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.990192 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.990146    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.990192 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.990191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77bw8\" (UniqueName: \"kubernetes.io/projected/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-kube-api-access-77bw8\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.990463 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.990260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-metrics-certs\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.990463 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:08:02.990296    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-serving-cert: secret "maas-api-e2e-mt-5c801c28-serving-cert" not found
Sep 10 15:08:02.990463 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.990279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-openshift-service-ca\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.990463 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:08:02.990376    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls podName:5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4 nodeName:}" failed. No retries permitted until 2026-09-10 15:08:03.490354713 +0000 UTC m=+1218.059893964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls") pod "maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg" (UID: "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4") : secret "maas-api-e2e-mt-5c801c28-serving-cert" not found
Sep 10 15:08:02.991005 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.990983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-openshift-service-ca\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:02.992864 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:02.992845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-metrics-certs\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:03.002171 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:03.002147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77bw8\" (UniqueName: \"kubernetes.io/projected/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-kube-api-access-77bw8\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:03.076484 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:03.076445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh" event={"ID":"7d57277e-4fc7-405a-a983-381b0a9291fe","Type":"ContainerStarted","Data":"276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781"}
Sep 10 15:08:03.097291 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:03.097238    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh" podStartSLOduration=2.097221078 podStartE2EDuration="2.097221078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:08:03.094495758 +0000 UTC m=+1217.664035017" watchObservedRunningTime="2026-09-10 15:08:03.097221078 +0000 UTC m=+1217.666760337"
Sep 10 15:08:03.494987 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:03.494953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:03.497271 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:03.497250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls\") pod \"maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:03.777398 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:03.777321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:04.056872 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:04.056844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"]
Sep 10 15:08:04.058224 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:08:04.058196    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdb01ef_fb4d_4e1e_ad8f_7d6f3376c7c4.slice/crio-4d1651750517dc91d61def382200ec439a48a38fec308fdc18e0027811a783f8 WatchSource:0}: Error finding container 4d1651750517dc91d61def382200ec439a48a38fec308fdc18e0027811a783f8: Status 404 returned error can't find the container with id 4d1651750517dc91d61def382200ec439a48a38fec308fdc18e0027811a783f8
Sep 10 15:08:04.082716 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:04.082683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg" event={"ID":"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4","Type":"ContainerStarted","Data":"4d1651750517dc91d61def382200ec439a48a38fec308fdc18e0027811a783f8"}
Sep 10 15:08:04.793254 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:04.793222    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:08:05.089928 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:05.089894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg" event={"ID":"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4","Type":"ContainerStarted","Data":"f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f"}
Sep 10 15:08:05.090315 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:05.089959    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:05.110840 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:05.110785    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg" podStartSLOduration=2.597746869 podStartE2EDuration="3.110769696s" totalImagesPullingTime="513.022827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:08:05.109220968 +0000 UTC m=+1219.678760228" watchObservedRunningTime="2026-09-10 15:08:05.110769696 +0000 UTC m=+1219.680308953"
Sep 10 15:08:11.100813 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:11.100783    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:08:11.767781 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:11.767741    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:11.767781 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:11.767792    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:11.780230 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:11.780204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:12.139953 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:12.139924    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:08:21.655740 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:21.655697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:08:34.271059 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:34.271019    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:08:41.858928 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.858833    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"]
Sep 10 15:08:41.873456 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.873422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:41.874413 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.874385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"]
Sep 10 15:08:41.877489 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.877461    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5c801c28\"/\"kube-root-ca.crt\""
Sep 10 15:08:41.877613 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.877462    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5c801c28\"/\"openshift-service-ca.crt\""
Sep 10 15:08:41.877703 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.877612    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5c801c28\"/\"e2e-lifecycle-model-5c801c28-kserve-self-signed-certs\""
Sep 10 15:08:41.877883 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.877868    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5c801c28\"/\"default-dockercfg-z46tv\""
Sep 10 15:08:41.959790 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.959760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-dshm\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:41.959983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.959800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2947x\" (UniqueName: \"kubernetes.io/projected/7654d295-5ddf-4e0c-99ed-1d723746ef6d-kube-api-access-2947x\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:41.959983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.959843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tmp-dir\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:41.959983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.959903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-home\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:41.959983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.959919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-model-cache\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:41.959983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:41.959949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tls-certs\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.060944 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.060910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-dshm\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061161 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.060954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2947x\" (UniqueName: \"kubernetes.io/projected/7654d295-5ddf-4e0c-99ed-1d723746ef6d-kube-api-access-2947x\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061161 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.060996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tmp-dir\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061161 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.061035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-home\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061161 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.061051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-model-cache\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061384 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.061217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tls-certs\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061454 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.061430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-model-cache\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061538 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.061514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tmp-dir\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.061597 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.061571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-home\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.063266 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.063247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-dshm\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.063741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.063723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tls-certs\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.070030 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.070004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2947x\" (UniqueName: \"kubernetes.io/projected/7654d295-5ddf-4e0c-99ed-1d723746ef6d-kube-api-access-2947x\") pod \"e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") " pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.183736 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.183601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:42.461832 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:42.461752    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"]
Sep 10 15:08:42.465055 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:08:42.465024    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7654d295_5ddf_4e0c_99ed_1d723746ef6d.slice/crio-2eafe2f55958671e696e73273aae03489e663f8e30a56f73f06b8e09849f302d WatchSource:0}: Error finding container 2eafe2f55958671e696e73273aae03489e663f8e30a56f73f06b8e09849f302d: Status 404 returned error can't find the container with id 2eafe2f55958671e696e73273aae03489e663f8e30a56f73f06b8e09849f302d
Sep 10 15:08:43.298922 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:43.298885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7" event={"ID":"7654d295-5ddf-4e0c-99ed-1d723746ef6d","Type":"ContainerStarted","Data":"cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c"}
Sep 10 15:08:43.299399 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:43.298934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7" event={"ID":"7654d295-5ddf-4e0c-99ed-1d723746ef6d","Type":"ContainerStarted","Data":"2eafe2f55958671e696e73273aae03489e663f8e30a56f73f06b8e09849f302d"}
Sep 10 15:08:43.323168 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:43.323117    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7" podStartSLOduration=2.323101071 podStartE2EDuration="2.323101071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:08:43.321503058 +0000 UTC m=+1257.891042316" watchObservedRunningTime="2026-09-10 15:08:43.323101071 +0000 UTC m=+1257.892640361"
Sep 10 15:08:52.165169 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:52.165135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:08:52.183918 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:52.183895    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:52.184070 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:52.183931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:52.196460 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:52.196437    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:08:52.357609 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:08:52.357580    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:09:10.255298 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:10.255267    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:09:26.647185 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.647151    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"]
Sep 10 15:09:26.647674 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.647401    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg" podUID="5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" containerName="maas-api" containerID="cri-o://f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f" gracePeriod=30
Sep 10 15:09:26.904409 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.904349    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:09:26.991135 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.991101    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77bw8\" (UniqueName: \"kubernetes.io/projected/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-kube-api-access-77bw8\") pod \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") "
Sep 10 15:09:26.991315 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.991156    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls\") pod \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") "
Sep 10 15:09:26.991315 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.991277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-openshift-service-ca\") pod \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") "
Sep 10 15:09:26.991433 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.991326    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-metrics-certs\") pod \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\" (UID: \"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4\") "
Sep 10 15:09:26.991680 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.991634    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" (UID: "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:09:26.993309 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.993283    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" (UID: "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:09:26.993419 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.993318    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" (UID: "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:09:26.993419 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:26.993320    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-kube-api-access-77bw8" (OuterVolumeSpecName: "kube-api-access-77bw8") pod "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" (UID: "5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4"). InnerVolumeSpecName "kube-api-access-77bw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:09:27.092781 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.092747    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77bw8\" (UniqueName: \"kubernetes.io/projected/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-kube-api-access-77bw8\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:27.092983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.092883    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:27.092983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.092908    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:27.092983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.092922    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:27.531224 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.531182    2565 generic.go:356] "Generic (PLEG): container finished" podID="5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" containerID="f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f" exitCode=0
Sep 10 15:09:27.531402 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.531236    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"
Sep 10 15:09:27.531402 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.531267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg" event={"ID":"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4","Type":"ContainerDied","Data":"f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f"}
Sep 10 15:09:27.531402 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.531323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg" event={"ID":"5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4","Type":"ContainerDied","Data":"4d1651750517dc91d61def382200ec439a48a38fec308fdc18e0027811a783f8"}
Sep 10 15:09:27.531402 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.531346    2565 scope.go:117] "RemoveContainer" containerID="f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f"
Sep 10 15:09:27.540819 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.540798    2565 scope.go:117] "RemoveContainer" containerID="f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f"
Sep 10 15:09:27.541141 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:09:27.541123    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f\": container with ID starting with f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f not found: ID does not exist" containerID="f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f"
Sep 10 15:09:27.541210 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.541152    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f"} err="failed to get container status \"f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f\": rpc error: code = NotFound desc = could not find container \"f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f\": container with ID starting with f91fca221aae14dcdb6079789527c01864adf26256d1e5478e1a7b2c0ad1c67f not found: ID does not exist"
Sep 10 15:09:27.553933 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.553905    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"]
Sep 10 15:09:27.557719 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.557691    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5c801c28-6c5dd9868d-j4ghg"]
Sep 10 15:09:27.983902 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:27.983865    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" path="/var/lib/kubelet/pods/5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4/volumes"
Sep 10 15:09:29.154455 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:29.154419    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:09:34.967332 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:34.967294    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"]
Sep 10 15:09:45.752532 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:45.752496    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:09:49.986985 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:49.986946    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7" podUID="7654d295-5ddf-4e0c-99ed-1d723746ef6d" containerName="main" containerID="cri-o://cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c" gracePeriod=45
Sep 10 15:09:50.235839 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.235809    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:09:50.401315 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-dshm\") pod \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") "
Sep 10 15:09:50.401315 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tmp-dir\") pod \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") "
Sep 10 15:09:50.401583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2947x\" (UniqueName: \"kubernetes.io/projected/7654d295-5ddf-4e0c-99ed-1d723746ef6d-kube-api-access-2947x\") pod \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") "
Sep 10 15:09:50.401583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-home\") pod \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") "
Sep 10 15:09:50.401583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401537    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-model-cache\") pod \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") "
Sep 10 15:09:50.401583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401561    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tls-certs\") pod \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\" (UID: \"7654d295-5ddf-4e0c-99ed-1d723746ef6d\") "
Sep 10 15:09:50.401823 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401613    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7654d295-5ddf-4e0c-99ed-1d723746ef6d" (UID: "7654d295-5ddf-4e0c-99ed-1d723746ef6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:50.401823 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-home" (OuterVolumeSpecName: "home") pod "7654d295-5ddf-4e0c-99ed-1d723746ef6d" (UID: "7654d295-5ddf-4e0c-99ed-1d723746ef6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:50.401939 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401819    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7654d295-5ddf-4e0c-99ed-1d723746ef6d" (UID: "7654d295-5ddf-4e0c-99ed-1d723746ef6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:50.401939 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401907    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:50.401939 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401918    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:50.401939 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.401928    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:50.403573 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.403541    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-dshm" (OuterVolumeSpecName: "dshm") pod "7654d295-5ddf-4e0c-99ed-1d723746ef6d" (UID: "7654d295-5ddf-4e0c-99ed-1d723746ef6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:50.403573 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.403563    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7654d295-5ddf-4e0c-99ed-1d723746ef6d-kube-api-access-2947x" (OuterVolumeSpecName: "kube-api-access-2947x") pod "7654d295-5ddf-4e0c-99ed-1d723746ef6d" (UID: "7654d295-5ddf-4e0c-99ed-1d723746ef6d"). InnerVolumeSpecName "kube-api-access-2947x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:09:50.403742 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.403628    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7654d295-5ddf-4e0c-99ed-1d723746ef6d" (UID: "7654d295-5ddf-4e0c-99ed-1d723746ef6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:09:50.503293 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.503254    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2947x\" (UniqueName: \"kubernetes.io/projected/7654d295-5ddf-4e0c-99ed-1d723746ef6d-kube-api-access-2947x\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:50.503293 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.503289    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7654d295-5ddf-4e0c-99ed-1d723746ef6d-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:50.503293 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.503301    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7654d295-5ddf-4e0c-99ed-1d723746ef6d-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:09:50.653014 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.652921    2565 generic.go:356] "Generic (PLEG): container finished" podID="7654d295-5ddf-4e0c-99ed-1d723746ef6d" containerID="cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c" exitCode=0
Sep 10 15:09:50.653014 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.652969    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7" event={"ID":"7654d295-5ddf-4e0c-99ed-1d723746ef6d","Type":"ContainerDied","Data":"cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c"}
Sep 10 15:09:50.653014 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.652972    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"
Sep 10 15:09:50.653014 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.653000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7" event={"ID":"7654d295-5ddf-4e0c-99ed-1d723746ef6d","Type":"ContainerDied","Data":"2eafe2f55958671e696e73273aae03489e663f8e30a56f73f06b8e09849f302d"}
Sep 10 15:09:50.653014 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.653019    2565 scope.go:117] "RemoveContainer" containerID="cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c"
Sep 10 15:09:50.663026 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.663007    2565 scope.go:117] "RemoveContainer" containerID="cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c"
Sep 10 15:09:50.663333 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:09:50.663313    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c\": container with ID starting with cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c not found: ID does not exist" containerID="cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c"
Sep 10 15:09:50.663404 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.663345    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c"} err="failed to get container status \"cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c\": rpc error: code = NotFound desc = could not find container \"cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c\": container with ID starting with cd70b23ea67d48f4ad8dea427eeccded49268fdfd139b8a0ec4b0684b5ad795c not found: ID does not exist"
Sep 10 15:09:50.676698 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.676671    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"]
Sep 10 15:09:50.679990 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:50.679965    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5c801c28/e2e-lifecycle-model-5c801c28-kserve-7fb8987547-98qm7"]
Sep 10 15:09:51.977172 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:09:51.977139    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7654d295-5ddf-4e0c-99ed-1d723746ef6d" path="/var/lib/kubelet/pods/7654d295-5ddf-4e0c-99ed-1d723746ef6d/volumes"
Sep 10 15:10:00.148527 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:00.148487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:10:03.127316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.127282    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-97066f"]
Sep 10 15:10:03.127811 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.127680    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" containerName="maas-api"
Sep 10 15:10:03.127811 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.127694    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" containerName="maas-api"
Sep 10 15:10:03.127811 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.127703    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7654d295-5ddf-4e0c-99ed-1d723746ef6d" containerName="main"
Sep 10 15:10:03.127811 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.127708    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7654d295-5ddf-4e0c-99ed-1d723746ef6d" containerName="main"
Sep 10 15:10:03.127811 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.127768    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7654d295-5ddf-4e0c-99ed-1d723746ef6d" containerName="main"
Sep 10 15:10:03.127811 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.127781    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fdb01ef-fb4d-4e1e-ad8f-7d6f3376c7c4" containerName="maas-api"
Sep 10 15:10:03.132236 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.132220    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:03.134732 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.134709    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrlvr\""
Sep 10 15:10:03.137715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.137678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-97066f"]
Sep 10 15:10:03.222832 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.222795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrls\" (UniqueName: \"kubernetes.io/projected/565a616d-13c8-4561-807a-b0a01406f290-kube-api-access-xbrls\") pod \"test-curl-7546-97066f\" (UID: \"565a616d-13c8-4561-807a-b0a01406f290\") " pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:03.323393 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.323359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrls\" (UniqueName: \"kubernetes.io/projected/565a616d-13c8-4561-807a-b0a01406f290-kube-api-access-xbrls\") pod \"test-curl-7546-97066f\" (UID: \"565a616d-13c8-4561-807a-b0a01406f290\") " pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:03.331840 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.331811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrls\" (UniqueName: \"kubernetes.io/projected/565a616d-13c8-4561-807a-b0a01406f290-kube-api-access-xbrls\") pod \"test-curl-7546-97066f\" (UID: \"565a616d-13c8-4561-807a-b0a01406f290\") " pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:03.443739 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.443630    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:03.720683 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:03.720654    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-97066f"]
Sep 10 15:10:03.722028 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:10:03.722000    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod565a616d_13c8_4561_807a_b0a01406f290.slice/crio-f10a7858e410f8f5ae9095939692bdd253c65ee48c5ca9225369051e60d9e662 WatchSource:0}: Error finding container f10a7858e410f8f5ae9095939692bdd253c65ee48c5ca9225369051e60d9e662: Status 404 returned error can't find the container with id f10a7858e410f8f5ae9095939692bdd253c65ee48c5ca9225369051e60d9e662
Sep 10 15:10:04.730421 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:04.730383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-97066f" event={"ID":"565a616d-13c8-4561-807a-b0a01406f290","Type":"ContainerStarted","Data":"8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376"}
Sep 10 15:10:04.730934 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:04.730433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-97066f" event={"ID":"565a616d-13c8-4561-807a-b0a01406f290","Type":"ContainerStarted","Data":"f10a7858e410f8f5ae9095939692bdd253c65ee48c5ca9225369051e60d9e662"}
Sep 10 15:10:04.745630 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:04.745581    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7546-97066f" podStartSLOduration=1.7455660229999999 podStartE2EDuration="1.745566023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:10:04.745189997 +0000 UTC m=+1339.314729256" watchObservedRunningTime="2026-09-10 15:10:04.745566023 +0000 UTC m=+1339.315105281"
Sep 10 15:10:05.328453 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:05.328416    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-97066f"]
Sep 10 15:10:05.332929 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:05.332898    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-97066f"]
Sep 10 15:10:05.735733 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:10:05.735703    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7546-97066f" container="test-curl-7546-97066f"
Sep 10 15:10:06.674924 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:06.674891    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"]
Sep 10 15:10:06.740754 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:06.740711    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7546-97066f" podUID="565a616d-13c8-4561-807a-b0a01406f290" containerName="test-curl-7546-97066f" containerID="cri-o://8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376" gracePeriod=2
Sep 10 15:10:07.741472 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:07.741426    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"]
Sep 10 15:10:08.883601 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:08.883574    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:08.971036 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:08.971004    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbrls\" (UniqueName: \"kubernetes.io/projected/565a616d-13c8-4561-807a-b0a01406f290-kube-api-access-xbrls\") pod \"565a616d-13c8-4561-807a-b0a01406f290\" (UID: \"565a616d-13c8-4561-807a-b0a01406f290\") "
Sep 10 15:10:08.972995 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:08.972964    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565a616d-13c8-4561-807a-b0a01406f290-kube-api-access-xbrls" (OuterVolumeSpecName: "kube-api-access-xbrls") pod "565a616d-13c8-4561-807a-b0a01406f290" (UID: "565a616d-13c8-4561-807a-b0a01406f290"). InnerVolumeSpecName "kube-api-access-xbrls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:09.072768 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:09.072666    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrls\" (UniqueName: \"kubernetes.io/projected/565a616d-13c8-4561-807a-b0a01406f290-kube-api-access-xbrls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:09.759083 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:09.759051    2565 generic.go:356] "Generic (PLEG): container finished" podID="565a616d-13c8-4561-807a-b0a01406f290" containerID="8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376" exitCode=137
Sep 10 15:10:09.759270 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:09.759110    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:09.759270 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:09.759158    2565 scope.go:117] "RemoveContainer" containerID="8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376"
Sep 10 15:10:09.768297 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:09.768279    2565 scope.go:117] "RemoveContainer" containerID="8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376"
Sep 10 15:10:09.768553 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:10:09.768533    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376\": container with ID starting with 8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376 not found: ID does not exist" containerID="8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376"
Sep 10 15:10:09.768594 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:09.768563    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376"} err="failed to get container status \"8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376\": rpc error: code = NotFound desc = could not find container \"8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376\": container with ID starting with 8840265f5626d4bf9dac31062e8292b6dfba342ac8828ba7304e817bfab1c376 not found: ID does not exist"
Sep 10 15:10:09.769206 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:10:09.769185    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-97066f\" is forbidden: User \"system:node:ip-10-0-130-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-2.ec2.internal' and this object" podUID="565a616d-13c8-4561-807a-b0a01406f290" pod="opendatahub/test-curl-7546-97066f"
Sep 10 15:10:09.977224 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:09.977197    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565a616d-13c8-4561-807a-b0a01406f290" path="/var/lib/kubelet/pods/565a616d-13c8-4561-807a-b0a01406f290/volumes"
Sep 10 15:10:17.855137 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:17.855097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:10:19.655417 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.655379    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"]
Sep 10 15:10:19.655938 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.655917    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565a616d-13c8-4561-807a-b0a01406f290" containerName="test-curl-7546-97066f"
Sep 10 15:10:19.656018 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.655940    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="565a616d-13c8-4561-807a-b0a01406f290" containerName="test-curl-7546-97066f"
Sep 10 15:10:19.656080 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.656068    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="565a616d-13c8-4561-807a-b0a01406f290" containerName="test-curl-7546-97066f"
Sep 10 15:10:19.659120 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.659100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.662051 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.662033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-decc52\"/\"sub-test-model-decc52-kserve-self-signed-certs\""
Sep 10 15:10:19.670513 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.670489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"]
Sep 10 15:10:19.778779 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.778743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-dshm\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.778977 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.778866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-model-cache\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.778977 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.778904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tmp-dir\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.778977 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.778921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tls-certs\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.778977 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.778943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-home\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.778977 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.778973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gkr\" (UniqueName: \"kubernetes.io/projected/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-kube-api-access-t2gkr\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.879943 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.879903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-model-cache\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.879954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tmp-dir\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.879978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tls-certs\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.880007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-home\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.880047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gkr\" (UniqueName: \"kubernetes.io/projected/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-kube-api-access-t2gkr\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880140 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.880111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-dshm\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.880380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tmp-dir\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.880388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-model-cache\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.880522 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.880447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-home\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.882322 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.882299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-dshm\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.882559 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.882544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tls-certs\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.890303 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.890284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gkr\" (UniqueName: \"kubernetes.io/projected/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-kube-api-access-t2gkr\") pod \"sub-test-model-decc52-kserve-77ff8fb587-mh6qq\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") " pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:19.967597 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:19.967517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:20.242179 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:20.242056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"]
Sep 10 15:10:20.244296 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:10:20.244270    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ba0a12_c2a3_467e_b1fc_d8e9600bc5c0.slice/crio-1b93783dc81054a6f214f8ce7c162908ed3efab8fbac242ea939b345e7ec2c4f WatchSource:0}: Error finding container 1b93783dc81054a6f214f8ce7c162908ed3efab8fbac242ea939b345e7ec2c4f: Status 404 returned error can't find the container with id 1b93783dc81054a6f214f8ce7c162908ed3efab8fbac242ea939b345e7ec2c4f
Sep 10 15:10:20.820711 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:20.820672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq" event={"ID":"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0","Type":"ContainerStarted","Data":"05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23"}
Sep 10 15:10:20.821087 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:20.820720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq" event={"ID":"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0","Type":"ContainerStarted","Data":"1b93783dc81054a6f214f8ce7c162908ed3efab8fbac242ea939b345e7ec2c4f"}
Sep 10 15:10:20.840070 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:20.840018    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq" podStartSLOduration=1.8400035479999999 podStartE2EDuration="1.840003548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:10:20.837787936 +0000 UTC m=+1355.407327194" watchObservedRunningTime="2026-09-10 15:10:20.840003548 +0000 UTC m=+1355.409542861"
Sep 10 15:10:21.694101 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:21.694055    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f" podUID="2292c02e-7e1c-458a-bc36-a92dae54ce08" containerName="main" containerID="cri-o://d4b49697f24dca816e813aa1759c9e894f2817ab74c7460efeb95b7e3f30892b" gracePeriod=45
Sep 10 15:10:21.828421 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:21.828384    2565 generic.go:356] "Generic (PLEG): container finished" podID="2292c02e-7e1c-458a-bc36-a92dae54ce08" containerID="d4b49697f24dca816e813aa1759c9e894f2817ab74c7460efeb95b7e3f30892b" exitCode=0
Sep 10 15:10:21.828795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:21.828420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f" event={"ID":"2292c02e-7e1c-458a-bc36-a92dae54ce08","Type":"ContainerDied","Data":"d4b49697f24dca816e813aa1759c9e894f2817ab74c7460efeb95b7e3f30892b"}
Sep 10 15:10:21.948067 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:21.948014    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:10:22.001307 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thp4n\" (UniqueName: \"kubernetes.io/projected/2292c02e-7e1c-458a-bc36-a92dae54ce08-kube-api-access-thp4n\") pod \"2292c02e-7e1c-458a-bc36-a92dae54ce08\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") "
Sep 10 15:10:22.001467 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001312    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2292c02e-7e1c-458a-bc36-a92dae54ce08-tls-certs\") pod \"2292c02e-7e1c-458a-bc36-a92dae54ce08\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") "
Sep 10 15:10:22.001467 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001352    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-model-cache\") pod \"2292c02e-7e1c-458a-bc36-a92dae54ce08\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") "
Sep 10 15:10:22.001467 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001381    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-tmp-dir\") pod \"2292c02e-7e1c-458a-bc36-a92dae54ce08\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") "
Sep 10 15:10:22.001467 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001404    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-home\") pod \"2292c02e-7e1c-458a-bc36-a92dae54ce08\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") "
Sep 10 15:10:22.001467 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001447    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-dshm\") pod \"2292c02e-7e1c-458a-bc36-a92dae54ce08\" (UID: \"2292c02e-7e1c-458a-bc36-a92dae54ce08\") "
Sep 10 15:10:22.001767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001620    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-model-cache" (OuterVolumeSpecName: "model-cache") pod "2292c02e-7e1c-458a-bc36-a92dae54ce08" (UID: "2292c02e-7e1c-458a-bc36-a92dae54ce08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:22.001767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-home" (OuterVolumeSpecName: "home") pod "2292c02e-7e1c-458a-bc36-a92dae54ce08" (UID: "2292c02e-7e1c-458a-bc36-a92dae54ce08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:22.001767 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2292c02e-7e1c-458a-bc36-a92dae54ce08" (UID: "2292c02e-7e1c-458a-bc36-a92dae54ce08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:22.001876 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001786    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:22.001876 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.001801    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:22.003613 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.003592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2292c02e-7e1c-458a-bc36-a92dae54ce08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2292c02e-7e1c-458a-bc36-a92dae54ce08" (UID: "2292c02e-7e1c-458a-bc36-a92dae54ce08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:10:22.003700 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.003625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-dshm" (OuterVolumeSpecName: "dshm") pod "2292c02e-7e1c-458a-bc36-a92dae54ce08" (UID: "2292c02e-7e1c-458a-bc36-a92dae54ce08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:22.003700 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.003666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2292c02e-7e1c-458a-bc36-a92dae54ce08-kube-api-access-thp4n" (OuterVolumeSpecName: "kube-api-access-thp4n") pod "2292c02e-7e1c-458a-bc36-a92dae54ce08" (UID: "2292c02e-7e1c-458a-bc36-a92dae54ce08"). InnerVolumeSpecName "kube-api-access-thp4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:22.102258 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.102222    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thp4n\" (UniqueName: \"kubernetes.io/projected/2292c02e-7e1c-458a-bc36-a92dae54ce08-kube-api-access-thp4n\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:22.102258 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.102254    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2292c02e-7e1c-458a-bc36-a92dae54ce08-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:22.102258 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.102267    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:22.102485 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.102278    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2292c02e-7e1c-458a-bc36-a92dae54ce08-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:22.761653 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.761601    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh" podUID="7d57277e-4fc7-405a-a983-381b0a9291fe" containerName="main" containerID="cri-o://276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781" gracePeriod=45
Sep 10 15:10:22.834603 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.834573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f" event={"ID":"2292c02e-7e1c-458a-bc36-a92dae54ce08","Type":"ContainerDied","Data":"e29abf05312a6134c336651b984f039cea319df474a6da42911147755f2bab08"}
Sep 10 15:10:22.834964 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.834618    2565 scope.go:117] "RemoveContainer" containerID="d4b49697f24dca816e813aa1759c9e894f2817ab74c7460efeb95b7e3f30892b"
Sep 10 15:10:22.834964 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.834581    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"
Sep 10 15:10:22.855379 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.855353    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"]
Sep 10 15:10:22.861050 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:22.861029    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/auth-test-model-decc52-kserve-5b5fcd76cf-hhn2f"]
Sep 10 15:10:23.005002 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.004977    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:10:23.112362 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112333    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d57277e-4fc7-405a-a983-381b0a9291fe-tls-certs\") pod \"7d57277e-4fc7-405a-a983-381b0a9291fe\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") "
Sep 10 15:10:23.112488 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112366    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-model-cache\") pod \"7d57277e-4fc7-405a-a983-381b0a9291fe\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") "
Sep 10 15:10:23.112488 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-tmp-dir\") pod \"7d57277e-4fc7-405a-a983-381b0a9291fe\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") "
Sep 10 15:10:23.112488 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112452    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-dshm\") pod \"7d57277e-4fc7-405a-a983-381b0a9291fe\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") "
Sep 10 15:10:23.112604 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112515    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tndm2\" (UniqueName: \"kubernetes.io/projected/7d57277e-4fc7-405a-a983-381b0a9291fe-kube-api-access-tndm2\") pod \"7d57277e-4fc7-405a-a983-381b0a9291fe\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") "
Sep 10 15:10:23.112604 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112557    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-home\") pod \"7d57277e-4fc7-405a-a983-381b0a9291fe\" (UID: \"7d57277e-4fc7-405a-a983-381b0a9291fe\") "
Sep 10 15:10:23.112759 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d57277e-4fc7-405a-a983-381b0a9291fe" (UID: "7d57277e-4fc7-405a-a983-381b0a9291fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:23.112822 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112739    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d57277e-4fc7-405a-a983-381b0a9291fe" (UID: "7d57277e-4fc7-405a-a983-381b0a9291fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:23.112906 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-home" (OuterVolumeSpecName: "home") pod "7d57277e-4fc7-405a-a983-381b0a9291fe" (UID: "7d57277e-4fc7-405a-a983-381b0a9291fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:23.112906 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112898    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:23.112998 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.112917    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:23.114666 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.114625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d57277e-4fc7-405a-a983-381b0a9291fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d57277e-4fc7-405a-a983-381b0a9291fe" (UID: "7d57277e-4fc7-405a-a983-381b0a9291fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:10:23.114758 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.114667    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-dshm" (OuterVolumeSpecName: "dshm") pod "7d57277e-4fc7-405a-a983-381b0a9291fe" (UID: "7d57277e-4fc7-405a-a983-381b0a9291fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:23.114758 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.114695    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d57277e-4fc7-405a-a983-381b0a9291fe-kube-api-access-tndm2" (OuterVolumeSpecName: "kube-api-access-tndm2") pod "7d57277e-4fc7-405a-a983-381b0a9291fe" (UID: "7d57277e-4fc7-405a-a983-381b0a9291fe"). InnerVolumeSpecName "kube-api-access-tndm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:23.214316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.214287    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:23.214316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.214310    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndm2\" (UniqueName: \"kubernetes.io/projected/7d57277e-4fc7-405a-a983-381b0a9291fe-kube-api-access-tndm2\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:23.214316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.214321    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d57277e-4fc7-405a-a983-381b0a9291fe-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:23.214503 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.214329    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d57277e-4fc7-405a-a983-381b0a9291fe-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:10:23.842039 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.842008    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d57277e-4fc7-405a-a983-381b0a9291fe" containerID="276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781" exitCode=0
Sep 10 15:10:23.842454 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.842051    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"
Sep 10 15:10:23.842454 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.842067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh" event={"ID":"7d57277e-4fc7-405a-a983-381b0a9291fe","Type":"ContainerDied","Data":"276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781"}
Sep 10 15:10:23.842454 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.842098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh" event={"ID":"7d57277e-4fc7-405a-a983-381b0a9291fe","Type":"ContainerDied","Data":"2ad101f3a179d42a0a0d865f177d3e46bf7f081d91736f43a2195bb2588be66f"}
Sep 10 15:10:23.842454 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.842116    2565 scope.go:117] "RemoveContainer" containerID="276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781"
Sep 10 15:10:23.851243 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.851226    2565 scope.go:117] "RemoveContainer" containerID="276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781"
Sep 10 15:10:23.851493 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:10:23.851476    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781\": container with ID starting with 276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781 not found: ID does not exist" containerID="276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781"
Sep 10 15:10:23.851553 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.851509    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781"} err="failed to get container status \"276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781\": rpc error: code = NotFound desc = could not find container \"276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781\": container with ID starting with 276daf5c55b24c92fc3df314ed019e01bfe16c67d0acb01ca66591802a3b7781 not found: ID does not exist"
Sep 10 15:10:23.863468 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.863436    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"]
Sep 10 15:10:23.866963 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.866941    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/auth-test-model-d5a2df-kserve-66b7fcddf9-7tmrh"]
Sep 10 15:10:23.977233 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.977201    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2292c02e-7e1c-458a-bc36-a92dae54ce08" path="/var/lib/kubelet/pods/2292c02e-7e1c-458a-bc36-a92dae54ce08/volumes"
Sep 10 15:10:23.977629 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:23.977608    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d57277e-4fc7-405a-a983-381b0a9291fe" path="/var/lib/kubelet/pods/7d57277e-4fc7-405a-a983-381b0a9291fe/volumes"
Sep 10 15:10:29.968299 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:29.968253    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:29.968695 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:29.968318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:29.981351 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:29.981324    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:30.890873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:30.890841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:10:31.559044 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:31.559009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:10:45.561857 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:10:45.561818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:11:23.182413 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.182377    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"]
Sep 10 15:11:23.182979 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.182962    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2292c02e-7e1c-458a-bc36-a92dae54ce08" containerName="main"
Sep 10 15:11:23.183025 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.182982    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2292c02e-7e1c-458a-bc36-a92dae54ce08" containerName="main"
Sep 10 15:11:23.183025 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.182998    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d57277e-4fc7-405a-a983-381b0a9291fe" containerName="main"
Sep 10 15:11:23.183025 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.183006    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d57277e-4fc7-405a-a983-381b0a9291fe" containerName="main"
Sep 10 15:11:23.183123 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.183079    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2292c02e-7e1c-458a-bc36-a92dae54ce08" containerName="main"
Sep 10 15:11:23.183123 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.183093    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d57277e-4fc7-405a-a983-381b0a9291fe" containerName="main"
Sep 10 15:11:23.186731 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.186708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.189049 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.189030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3e7a73-serving-cert\""
Sep 10 15:11:23.194350 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.194328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"]
Sep 10 15:11:23.280442 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.280412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.280597 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.280474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4qc\" (UniqueName: \"kubernetes.io/projected/b74cd979-a513-4767-96be-0d0af83329f7-kube-api-access-kh4qc\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.280597 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.280549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-metrics-certs\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.280750 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.280613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74cd979-a513-4767-96be-0d0af83329f7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.381828 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.381801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74cd979-a513-4767-96be-0d0af83329f7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.381992 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.381854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.381992 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:11:23.381959    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-serving-cert: secret "maas-api-e2e-api-a-3e7a73-serving-cert" not found
Sep 10 15:11:23.381992 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.381969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4qc\" (UniqueName: \"kubernetes.io/projected/b74cd979-a513-4767-96be-0d0af83329f7-kube-api-access-kh4qc\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.382189 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:11:23.382033    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls podName:b74cd979-a513-4767-96be-0d0af83329f7 nodeName:}" failed. No retries permitted until 2026-09-10 15:11:23.882010313 +0000 UTC m=+1418.451549564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls") pod "maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z" (UID: "b74cd979-a513-4767-96be-0d0af83329f7") : secret "maas-api-e2e-api-a-3e7a73-serving-cert" not found
Sep 10 15:11:23.382189 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.382050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-metrics-certs\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.382564 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.382540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74cd979-a513-4767-96be-0d0af83329f7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.384489 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.384471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-metrics-certs\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.393008 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.392984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4qc\" (UniqueName: \"kubernetes.io/projected/b74cd979-a513-4767-96be-0d0af83329f7-kube-api-access-kh4qc\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.887093 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.887060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:23.889359 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:23.889337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:24.099314 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:24.099281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:24.374798 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:24.374770    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"]
Sep 10 15:11:24.375915 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:11:24.375886    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74cd979_a513_4767_96be_0d0af83329f7.slice/crio-17ee3bbc4ff9c8ad007988b03c25fe221ee8fdd9c0dff0c85723e52f75ce4297 WatchSource:0}: Error finding container 17ee3bbc4ff9c8ad007988b03c25fe221ee8fdd9c0dff0c85723e52f75ce4297: Status 404 returned error can't find the container with id 17ee3bbc4ff9c8ad007988b03c25fe221ee8fdd9c0dff0c85723e52f75ce4297
Sep 10 15:11:25.163012 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:25.162926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z" event={"ID":"b74cd979-a513-4767-96be-0d0af83329f7","Type":"ContainerStarted","Data":"e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd"}
Sep 10 15:11:25.163012 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:25.162982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:25.163012 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:25.163004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z" event={"ID":"b74cd979-a513-4767-96be-0d0af83329f7","Type":"ContainerStarted","Data":"17ee3bbc4ff9c8ad007988b03c25fe221ee8fdd9c0dff0c85723e52f75ce4297"}
Sep 10 15:11:25.181022 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:25.180974    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z" podStartSLOduration=1.708971885 podStartE2EDuration="2.180959386s" totalImagesPullingTime="471.987501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:25.180099298 +0000 UTC m=+1419.749638556" watchObservedRunningTime="2026-09-10 15:11:25.180959386 +0000 UTC m=+1419.750498643"
Sep 10 15:11:29.461623 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.461587    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"]
Sep 10 15:11:29.465220 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.465203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.467757 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.467733    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"openshift-service-ca.crt\""
Sep 10 15:11:29.468689 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.468671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"sub-test-model-d5a2df-kserve-self-signed-certs\""
Sep 10 15:11:29.468795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.468692    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"kube-root-ca.crt\""
Sep 10 15:11:29.468795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.468700    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"default-dockercfg-wfhrm\""
Sep 10 15:11:29.474521 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.474499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"]
Sep 10 15:11:29.540147 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.540114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-model-cache\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.540147 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.540158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tls-certs\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.540369 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.540187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-dshm\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.540369 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.540261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tmp-dir\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.540369 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.540301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-home\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.540369 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.540319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6qf\" (UniqueName: \"kubernetes.io/projected/098cd6f6-04c6-403f-94ec-901d5b5bbb13-kube-api-access-hj6qf\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.641710 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.641665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-model-cache\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.641710 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.641717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tls-certs\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.641968 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.641843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-dshm\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.641968 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.641950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tmp-dir\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.642085 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.641989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-home\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.642085 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.642014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6qf\" (UniqueName: \"kubernetes.io/projected/098cd6f6-04c6-403f-94ec-901d5b5bbb13-kube-api-access-hj6qf\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.642187 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.642139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-model-cache\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.642338 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.642309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tmp-dir\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.642421 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.642316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-home\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.644172 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.644145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-dshm\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.644408 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.644390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tls-certs\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.651102 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.651079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6qf\" (UniqueName: \"kubernetes.io/projected/098cd6f6-04c6-403f-94ec-901d5b5bbb13-kube-api-access-hj6qf\") pod \"sub-test-model-d5a2df-kserve-96dbc587-gtn6r\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:29.775662 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:29.775549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:30.047108 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:30.047081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"]
Sep 10 15:11:30.048303 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:11:30.048277    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod098cd6f6_04c6_403f_94ec_901d5b5bbb13.slice/crio-2568611c7dce70dbdb9db96cf5ea4730c7e8286d73df324456a5ff9ec0722b0b WatchSource:0}: Error finding container 2568611c7dce70dbdb9db96cf5ea4730c7e8286d73df324456a5ff9ec0722b0b: Status 404 returned error can't find the container with id 2568611c7dce70dbdb9db96cf5ea4730c7e8286d73df324456a5ff9ec0722b0b
Sep 10 15:11:30.190421 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:30.190381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r" event={"ID":"098cd6f6-04c6-403f-94ec-901d5b5bbb13","Type":"ContainerStarted","Data":"617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a"}
Sep 10 15:11:30.190594 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:30.190433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r" event={"ID":"098cd6f6-04c6-403f-94ec-901d5b5bbb13","Type":"ContainerStarted","Data":"2568611c7dce70dbdb9db96cf5ea4730c7e8286d73df324456a5ff9ec0722b0b"}
Sep 10 15:11:30.207888 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:30.207835    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r" podStartSLOduration=1.207817988 podStartE2EDuration="1.207817988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:30.207298382 +0000 UTC m=+1424.776837641" watchObservedRunningTime="2026-09-10 15:11:30.207817988 +0000 UTC m=+1424.777357268"
Sep 10 15:11:31.173034 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:31.173006    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:11:33.867551 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:33.867504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:11:39.776390 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:39.776306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:39.776840 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:39.776387    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:39.789003 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:39.788978    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:40.259956 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:40.259930    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:11:52.260820 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.260785    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"]
Sep 10 15:11:52.264394 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.264375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.267751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.267729    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d7aa05-serving-cert\""
Sep 10 15:11:52.279585 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.279563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"]
Sep 10 15:11:52.347069 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.347035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-metrics-certs\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.347069 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.347071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca069e-e2be-4cf4-b99e-546c322f0c7d-openshift-service-ca\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.347281 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.347166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tztzz\" (UniqueName: \"kubernetes.io/projected/5cca069e-e2be-4cf4-b99e-546c322f0c7d-kube-api-access-tztzz\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.347281 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.347275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-maas-api-tls\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.448276 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.448239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-maas-api-tls\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.448276 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.448280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-metrics-certs\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.448535 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.448299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca069e-e2be-4cf4-b99e-546c322f0c7d-openshift-service-ca\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.448535 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.448344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tztzz\" (UniqueName: \"kubernetes.io/projected/5cca069e-e2be-4cf4-b99e-546c322f0c7d-kube-api-access-tztzz\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.449066 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.449038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca069e-e2be-4cf4-b99e-546c322f0c7d-openshift-service-ca\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.450677 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.450655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-maas-api-tls\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.450820 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.450798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-metrics-certs\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.456265 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.456242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tztzz\" (UniqueName: \"kubernetes.io/projected/5cca069e-e2be-4cf4-b99e-546c322f0c7d-kube-api-access-tztzz\") pod \"maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.575136 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.575039    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:52.851473 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.851440    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"]
Sep 10 15:11:52.853047 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:11:52.853020    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cca069e_e2be_4cf4_b99e_546c322f0c7d.slice/crio-af35c0789b3e993234c64dd36754cfac71b7aa96430b18d1a5a47e7ff61d24fe WatchSource:0}: Error finding container af35c0789b3e993234c64dd36754cfac71b7aa96430b18d1a5a47e7ff61d24fe: Status 404 returned error can't find the container with id af35c0789b3e993234c64dd36754cfac71b7aa96430b18d1a5a47e7ff61d24fe
Sep 10 15:11:52.854868 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:52.854851    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:11:53.320901 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:53.320802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk" event={"ID":"5cca069e-e2be-4cf4-b99e-546c322f0c7d","Type":"ContainerStarted","Data":"af35c0789b3e993234c64dd36754cfac71b7aa96430b18d1a5a47e7ff61d24fe"}
Sep 10 15:11:54.328751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:54.328714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk" event={"ID":"5cca069e-e2be-4cf4-b99e-546c322f0c7d","Type":"ContainerStarted","Data":"f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6"}
Sep 10 15:11:54.329313 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:54.328777    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:11:54.361041 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:11:54.360991    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk" podStartSLOduration=1.702415147 podStartE2EDuration="2.360975022s" totalImagesPullingTime="658.559875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:11:54.357594197 +0000 UTC m=+1448.927133454" watchObservedRunningTime="2026-09-10 15:11:54.360975022 +0000 UTC m=+1448.930514281"
Sep 10 15:12:00.340208 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:00.340178    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:12:03.458165 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:03.458129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:12:23.355680 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:23.355653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:12:26.482470 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.482435    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-c85e8f"]
Sep 10 15:12:26.486333 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.486313    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:26.488763 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.488742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrlvr\""
Sep 10 15:12:26.492594 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.492570    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-c85e8f"]
Sep 10 15:12:26.554471 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.554442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbsn2\" (UniqueName: \"kubernetes.io/projected/fa7e54d9-4259-45fd-aed4-32f5c845f9ff-kube-api-access-mbsn2\") pod \"test-curl-7546-c85e8f\" (UID: \"fa7e54d9-4259-45fd-aed4-32f5c845f9ff\") " pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:26.655889 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.655836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbsn2\" (UniqueName: \"kubernetes.io/projected/fa7e54d9-4259-45fd-aed4-32f5c845f9ff-kube-api-access-mbsn2\") pod \"test-curl-7546-c85e8f\" (UID: \"fa7e54d9-4259-45fd-aed4-32f5c845f9ff\") " pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:26.666408 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.666384    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbsn2\" (UniqueName: \"kubernetes.io/projected/fa7e54d9-4259-45fd-aed4-32f5c845f9ff-kube-api-access-mbsn2\") pod \"test-curl-7546-c85e8f\" (UID: \"fa7e54d9-4259-45fd-aed4-32f5c845f9ff\") " pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:26.797685 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:26.797584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:27.069301 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:27.069270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-c85e8f"]
Sep 10 15:12:27.070590 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:12:27.070560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa7e54d9_4259_45fd_aed4_32f5c845f9ff.slice/crio-be88c3d42dcbe88d6eca59b864d9f38ed8e8c8185de6d4a8381076626f278fca WatchSource:0}: Error finding container be88c3d42dcbe88d6eca59b864d9f38ed8e8c8185de6d4a8381076626f278fca: Status 404 returned error can't find the container with id be88c3d42dcbe88d6eca59b864d9f38ed8e8c8185de6d4a8381076626f278fca
Sep 10 15:12:27.502545 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:27.502509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-c85e8f" event={"ID":"fa7e54d9-4259-45fd-aed4-32f5c845f9ff","Type":"ContainerStarted","Data":"6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424"}
Sep 10 15:12:27.502545 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:27.502553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-c85e8f" event={"ID":"fa7e54d9-4259-45fd-aed4-32f5c845f9ff","Type":"ContainerStarted","Data":"be88c3d42dcbe88d6eca59b864d9f38ed8e8c8185de6d4a8381076626f278fca"}
Sep 10 15:12:27.518448 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:27.518400    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7546-c85e8f" podStartSLOduration=1.518383796 podStartE2EDuration="1.518383796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:12:27.515750525 +0000 UTC m=+1482.085289784" watchObservedRunningTime="2026-09-10 15:12:27.518383796 +0000 UTC m=+1482.087923053"
Sep 10 15:12:28.098896 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.098863    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-c85e8f"]
Sep 10 15:12:28.105795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.105763    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-c85e8f"]
Sep 10 15:12:28.217854 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.217773    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-02c3fd"]
Sep 10 15:12:28.218168 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.218154    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" containerName="test-curl-7546-c85e8f"
Sep 10 15:12:28.218224 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.218170    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" containerName="test-curl-7546-c85e8f"
Sep 10 15:12:28.218269 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.218242    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" containerName="test-curl-7546-c85e8f"
Sep 10 15:12:28.221206 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.221190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:28.227735 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.227706    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-02c3fd"]
Sep 10 15:12:28.270591 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.270553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhvs9\" (UniqueName: \"kubernetes.io/projected/939f980d-88e9-42b1-90ad-59c6aa34041e-kube-api-access-bhvs9\") pod \"test-curl-7546-02c3fd\" (UID: \"939f980d-88e9-42b1-90ad-59c6aa34041e\") " pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:28.371936 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.371900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhvs9\" (UniqueName: \"kubernetes.io/projected/939f980d-88e9-42b1-90ad-59c6aa34041e-kube-api-access-bhvs9\") pod \"test-curl-7546-02c3fd\" (UID: \"939f980d-88e9-42b1-90ad-59c6aa34041e\") " pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:28.381253 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.381227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhvs9\" (UniqueName: \"kubernetes.io/projected/939f980d-88e9-42b1-90ad-59c6aa34041e-kube-api-access-bhvs9\") pod \"test-curl-7546-02c3fd\" (UID: \"939f980d-88e9-42b1-90ad-59c6aa34041e\") " pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:28.507565 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:28.507477    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7546-c85e8f" container="test-curl-7546-c85e8f"
Sep 10 15:12:28.531949 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.531929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:28.803501 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:28.803473    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-02c3fd"]
Sep 10 15:12:28.805001 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:12:28.804970    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod939f980d_88e9_42b1_90ad_59c6aa34041e.slice/crio-dcf6063bc0f3ad18b41e47328b6774741a6dd22e6e6c5ecb24bc30463869cabf WatchSource:0}: Error finding container dcf6063bc0f3ad18b41e47328b6774741a6dd22e6e6c5ecb24bc30463869cabf: Status 404 returned error can't find the container with id dcf6063bc0f3ad18b41e47328b6774741a6dd22e6e6c5ecb24bc30463869cabf
Sep 10 15:12:29.514721 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:29.514685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-02c3fd" event={"ID":"939f980d-88e9-42b1-90ad-59c6aa34041e","Type":"ContainerStarted","Data":"6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29"}
Sep 10 15:12:29.515175 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:29.514729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-02c3fd" event={"ID":"939f980d-88e9-42b1-90ad-59c6aa34041e","Type":"ContainerStarted","Data":"dcf6063bc0f3ad18b41e47328b6774741a6dd22e6e6c5ecb24bc30463869cabf"}
Sep 10 15:12:29.515175 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:29.514805    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7546-c85e8f" podUID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" containerName="test-curl-7546-c85e8f" containerID="cri-o://6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424" gracePeriod=2
Sep 10 15:12:29.531861 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:29.531811    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7546-02c3fd" podStartSLOduration=1.53179609 podStartE2EDuration="1.53179609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:12:29.53095651 +0000 UTC m=+1484.100495767" watchObservedRunningTime="2026-09-10 15:12:29.53179609 +0000 UTC m=+1484.101335350"
Sep 10 15:12:29.533240 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:29.533212    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-c85e8f\" is forbidden: User \"system:node:ip-10-0-130-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-2.ec2.internal' and this object" podUID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:29.535280 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:29.535253    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-c85e8f\" is forbidden: User \"system:node:ip-10-0-130-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-2.ec2.internal' and this object" podUID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:30.126086 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:30.126049    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-02c3fd"]
Sep 10 15:12:30.129685 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:30.129628    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-02c3fd"]
Sep 10 15:12:30.519777 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:30.519739    2565 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7546-02c3fd" container="test-curl-7546-02c3fd"
Sep 10 15:12:31.527221 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:31.527180    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7546-02c3fd" podUID="939f980d-88e9-42b1-90ad-59c6aa34041e" containerName="test-curl-7546-02c3fd" containerID="cri-o://6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29" gracePeriod=2
Sep 10 15:12:31.529434 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:31.529402    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-02c3fd\" is forbidden: User \"system:node:ip-10-0-130-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-2.ec2.internal' and this object" podUID="939f980d-88e9-42b1-90ad-59c6aa34041e" pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:31.656452 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:31.656429    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:31.659068 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:31.659042    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-02c3fd\" is forbidden: User \"system:node:ip-10-0-130-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-2.ec2.internal' and this object" podUID="939f980d-88e9-42b1-90ad-59c6aa34041e" pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:31.806172 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:31.806091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbsn2\" (UniqueName: \"kubernetes.io/projected/fa7e54d9-4259-45fd-aed4-32f5c845f9ff-kube-api-access-mbsn2\") pod \"fa7e54d9-4259-45fd-aed4-32f5c845f9ff\" (UID: \"fa7e54d9-4259-45fd-aed4-32f5c845f9ff\") "
Sep 10 15:12:31.808115 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:31.808086    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7e54d9-4259-45fd-aed4-32f5c845f9ff-kube-api-access-mbsn2" (OuterVolumeSpecName: "kube-api-access-mbsn2") pod "fa7e54d9-4259-45fd-aed4-32f5c845f9ff" (UID: "fa7e54d9-4259-45fd-aed4-32f5c845f9ff"). InnerVolumeSpecName "kube-api-access-mbsn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:12:31.907583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:31.907546    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbsn2\" (UniqueName: \"kubernetes.io/projected/fa7e54d9-4259-45fd-aed4-32f5c845f9ff-kube-api-access-mbsn2\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:31.977945 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:31.977912    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" path="/var/lib/kubelet/pods/fa7e54d9-4259-45fd-aed4-32f5c845f9ff/volumes"
Sep 10 15:12:32.056576 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:32.056495    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"]
Sep 10 15:12:32.533411 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:32.533377    2565 generic.go:356] "Generic (PLEG): container finished" podID="fa7e54d9-4259-45fd-aed4-32f5c845f9ff" containerID="6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424" exitCode=137
Sep 10 15:12:32.533858 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:32.533423    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-c85e8f"
Sep 10 15:12:32.533858 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:32.533477    2565 scope.go:117] "RemoveContainer" containerID="6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424"
Sep 10 15:12:32.542430 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:32.542412    2565 scope.go:117] "RemoveContainer" containerID="6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424"
Sep 10 15:12:32.542740 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:32.542722    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424\": container with ID starting with 6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424 not found: ID does not exist" containerID="6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424"
Sep 10 15:12:32.542795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:32.542751    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424"} err="failed to get container status \"6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424\": rpc error: code = NotFound desc = could not find container \"6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424\": container with ID starting with 6f54a18db1432ad52cd27587213590f1745e67455cbae072910cc9c5247fd424 not found: ID does not exist"
Sep 10 15:12:33.049861 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:33.049820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"]
Sep 10 15:12:33.703126 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:33.703100    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:33.833131 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:33.833051    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhvs9\" (UniqueName: \"kubernetes.io/projected/939f980d-88e9-42b1-90ad-59c6aa34041e-kube-api-access-bhvs9\") pod \"939f980d-88e9-42b1-90ad-59c6aa34041e\" (UID: \"939f980d-88e9-42b1-90ad-59c6aa34041e\") "
Sep 10 15:12:33.835076 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:33.835046    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939f980d-88e9-42b1-90ad-59c6aa34041e-kube-api-access-bhvs9" (OuterVolumeSpecName: "kube-api-access-bhvs9") pod "939f980d-88e9-42b1-90ad-59c6aa34041e" (UID: "939f980d-88e9-42b1-90ad-59c6aa34041e"). InnerVolumeSpecName "kube-api-access-bhvs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:12:33.934570 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:33.934531    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhvs9\" (UniqueName: \"kubernetes.io/projected/939f980d-88e9-42b1-90ad-59c6aa34041e-kube-api-access-bhvs9\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:33.976966 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:33.976929    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939f980d-88e9-42b1-90ad-59c6aa34041e" path="/var/lib/kubelet/pods/939f980d-88e9-42b1-90ad-59c6aa34041e/volumes"
Sep 10 15:12:34.464664 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.464618    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"]
Sep 10 15:12:34.464858 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.464828    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z" podUID="b74cd979-a513-4767-96be-0d0af83329f7" containerName="maas-api" containerID="cri-o://e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd" gracePeriod=30
Sep 10 15:12:34.548721 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.548688    2565 generic.go:356] "Generic (PLEG): container finished" podID="939f980d-88e9-42b1-90ad-59c6aa34041e" containerID="6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29" exitCode=137
Sep 10 15:12:34.548912 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.548750    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-02c3fd"
Sep 10 15:12:34.548912 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.548795    2565 scope.go:117] "RemoveContainer" containerID="6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29"
Sep 10 15:12:34.576482 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.576349    2565 scope.go:117] "RemoveContainer" containerID="6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29"
Sep 10 15:12:34.576728 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:34.576698    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29\": container with ID starting with 6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29 not found: ID does not exist" containerID="6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29"
Sep 10 15:12:34.576834 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.576731    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29"} err="failed to get container status \"6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29\": rpc error: code = NotFound desc = could not find container \"6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29\": container with ID starting with 6d34e302f52baac4ebb06af0e4286b2d8e0260b14e9eec451f7d880cd1d2bc29 not found: ID does not exist"
Sep 10 15:12:34.717029 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.716961    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:12:34.844067 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.844037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-metrics-certs\") pod \"b74cd979-a513-4767-96be-0d0af83329f7\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") "
Sep 10 15:12:34.844223 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.844112    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls\") pod \"b74cd979-a513-4767-96be-0d0af83329f7\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") "
Sep 10 15:12:34.844223 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.844171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh4qc\" (UniqueName: \"kubernetes.io/projected/b74cd979-a513-4767-96be-0d0af83329f7-kube-api-access-kh4qc\") pod \"b74cd979-a513-4767-96be-0d0af83329f7\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") "
Sep 10 15:12:34.844223 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.844213    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74cd979-a513-4767-96be-0d0af83329f7-openshift-service-ca\") pod \"b74cd979-a513-4767-96be-0d0af83329f7\" (UID: \"b74cd979-a513-4767-96be-0d0af83329f7\") "
Sep 10 15:12:34.844664 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.844615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74cd979-a513-4767-96be-0d0af83329f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b74cd979-a513-4767-96be-0d0af83329f7" (UID: "b74cd979-a513-4767-96be-0d0af83329f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:12:34.846098 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.846075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b74cd979-a513-4767-96be-0d0af83329f7" (UID: "b74cd979-a513-4767-96be-0d0af83329f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:12:34.846209 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.846135    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74cd979-a513-4767-96be-0d0af83329f7-kube-api-access-kh4qc" (OuterVolumeSpecName: "kube-api-access-kh4qc") pod "b74cd979-a513-4767-96be-0d0af83329f7" (UID: "b74cd979-a513-4767-96be-0d0af83329f7"). InnerVolumeSpecName "kube-api-access-kh4qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:12:34.846209 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.846203    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b74cd979-a513-4767-96be-0d0af83329f7" (UID: "b74cd979-a513-4767-96be-0d0af83329f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:12:34.945522 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.945490    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:34.945522 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.945516    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4qc\" (UniqueName: \"kubernetes.io/projected/b74cd979-a513-4767-96be-0d0af83329f7-kube-api-access-kh4qc\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:34.945522 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.945527    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74cd979-a513-4767-96be-0d0af83329f7-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:34.945777 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:34.945536    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74cd979-a513-4767-96be-0d0af83329f7-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:35.555820 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.555786    2565 generic.go:356] "Generic (PLEG): container finished" podID="b74cd979-a513-4767-96be-0d0af83329f7" containerID="e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd" exitCode=0
Sep 10 15:12:35.556022 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.555839    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"
Sep 10 15:12:35.556022 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.555865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z" event={"ID":"b74cd979-a513-4767-96be-0d0af83329f7","Type":"ContainerDied","Data":"e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd"}
Sep 10 15:12:35.556022 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.555920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z" event={"ID":"b74cd979-a513-4767-96be-0d0af83329f7","Type":"ContainerDied","Data":"17ee3bbc4ff9c8ad007988b03c25fe221ee8fdd9c0dff0c85723e52f75ce4297"}
Sep 10 15:12:35.556022 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.555952    2565 scope.go:117] "RemoveContainer" containerID="e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd"
Sep 10 15:12:35.565511 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.565495    2565 scope.go:117] "RemoveContainer" containerID="e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd"
Sep 10 15:12:35.565776 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:35.565758    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd\": container with ID starting with e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd not found: ID does not exist" containerID="e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd"
Sep 10 15:12:35.565829 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.565783    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd"} err="failed to get container status \"e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd\": rpc error: code = NotFound desc = could not find container \"e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd\": container with ID starting with e39f64c9856f87c68c8010745faaa548c12f4e85d8d461806bc80cd4ae02fadd not found: ID does not exist"
Sep 10 15:12:35.585287 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.585266    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"]
Sep 10 15:12:35.590983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.590962    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7a73-668cc77df5-xdc7z"]
Sep 10 15:12:35.977328 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:35.977297    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74cd979-a513-4767-96be-0d0af83329f7" path="/var/lib/kubelet/pods/b74cd979-a513-4767-96be-0d0af83329f7/volumes"
Sep 10 15:12:39.259222 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:39.259180    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:12:40.561748 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.561712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"]
Sep 10 15:12:40.562120 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.562062    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939f980d-88e9-42b1-90ad-59c6aa34041e" containerName="test-curl-7546-02c3fd"
Sep 10 15:12:40.562120 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.562075    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="939f980d-88e9-42b1-90ad-59c6aa34041e" containerName="test-curl-7546-02c3fd"
Sep 10 15:12:40.562120 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.562097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74cd979-a513-4767-96be-0d0af83329f7" containerName="maas-api"
Sep 10 15:12:40.562120 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.562102    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74cd979-a513-4767-96be-0d0af83329f7" containerName="maas-api"
Sep 10 15:12:40.562256 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.562157    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="939f980d-88e9-42b1-90ad-59c6aa34041e" containerName="test-curl-7546-02c3fd"
Sep 10 15:12:40.562256 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.562165    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74cd979-a513-4767-96be-0d0af83329f7" containerName="maas-api"
Sep 10 15:12:40.566538 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.566520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.568963 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.568933    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-decc52\"/\"rate-test-model-decc52-kserve-self-signed-certs\""
Sep 10 15:12:40.574740 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.574327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"]
Sep 10 15:12:40.701771 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.701737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-tmp-dir\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.701968 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.701787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-home\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.701968 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.701848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9b8\" (UniqueName: \"kubernetes.io/projected/6b122d09-7226-403f-b3df-8e8944afb467-kube-api-access-ld9b8\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.701968 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.701892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-dshm\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.701968 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.701929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b122d09-7226-403f-b3df-8e8944afb467-tls-certs\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.702146 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.702006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-model-cache\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.802936 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.802886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-dshm\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.802936 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.802948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b122d09-7226-403f-b3df-8e8944afb467-tls-certs\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.803176 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.803004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-model-cache\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.803176 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.803032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-tmp-dir\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.803176 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.803057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-home\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.803176 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.803085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9b8\" (UniqueName: \"kubernetes.io/projected/6b122d09-7226-403f-b3df-8e8944afb467-kube-api-access-ld9b8\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.803462 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.803438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-model-cache\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.803539 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.803474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-tmp-dir\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.803595 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.803543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-home\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.805347 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.805321    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-dshm\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.805476 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.805408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b122d09-7226-403f-b3df-8e8944afb467-tls-certs\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.814988 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.814931    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9b8\" (UniqueName: \"kubernetes.io/projected/6b122d09-7226-403f-b3df-8e8944afb467-kube-api-access-ld9b8\") pod \"rate-test-model-decc52-kserve-7bdc95dc79-5d8dt\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") " pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:40.877085 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:40.877051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:41.149776 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:41.149751    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"]
Sep 10 15:12:41.152264 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:12:41.152230    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b122d09_7226_403f_b3df_8e8944afb467.slice/crio-9dbb6b155b66abd09c8e0a4bc87a9ba972165017c27598a812bbf747f7023d9d WatchSource:0}: Error finding container 9dbb6b155b66abd09c8e0a4bc87a9ba972165017c27598a812bbf747f7023d9d: Status 404 returned error can't find the container with id 9dbb6b155b66abd09c8e0a4bc87a9ba972165017c27598a812bbf747f7023d9d
Sep 10 15:12:41.592426 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:41.592334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt" event={"ID":"6b122d09-7226-403f-b3df-8e8944afb467","Type":"ContainerStarted","Data":"0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52"}
Sep 10 15:12:41.592426 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:41.592378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt" event={"ID":"6b122d09-7226-403f-b3df-8e8944afb467","Type":"ContainerStarted","Data":"9dbb6b155b66abd09c8e0a4bc87a9ba972165017c27598a812bbf747f7023d9d"}
Sep 10 15:12:41.610290 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:41.610240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt" podStartSLOduration=1.6102265500000001 podStartE2EDuration="1.61022655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:12:41.608845011 +0000 UTC m=+1496.178384271" watchObservedRunningTime="2026-09-10 15:12:41.61022655 +0000 UTC m=+1496.179765808"
Sep 10 15:12:47.076626 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.076572    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq" podUID="b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" containerName="main" containerID="cri-o://05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23" gracePeriod=45
Sep 10 15:12:47.316818 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.316794    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:12:47.363560 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-model-cache\") pod \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") "
Sep 10 15:12:47.363730 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tls-certs\") pod \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") "
Sep 10 15:12:47.363730 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363593    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-home\") pod \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") "
Sep 10 15:12:47.363865 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363743    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tmp-dir\") pod \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") "
Sep 10 15:12:47.363865 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2gkr\" (UniqueName: \"kubernetes.io/projected/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-kube-api-access-t2gkr\") pod \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") "
Sep 10 15:12:47.363865 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363818    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-home" (OuterVolumeSpecName: "home") pod "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" (UID: "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:47.363865 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" (UID: "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:47.363865 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363847    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-dshm\") pod \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\" (UID: \"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0\") "
Sep 10 15:12:47.364114 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.363970    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" (UID: "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:47.364246 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.364229    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:47.364307 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.364250    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:47.364307 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.364263    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:47.365753 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.365731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-dshm" (OuterVolumeSpecName: "dshm") pod "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" (UID: "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:47.366141 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.366119    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-kube-api-access-t2gkr" (OuterVolumeSpecName: "kube-api-access-t2gkr") pod "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" (UID: "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0"). InnerVolumeSpecName "kube-api-access-t2gkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:12:47.366233 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.366120    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" (UID: "b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:12:47.465098 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.465062    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:47.465098 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.465094    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2gkr\" (UniqueName: \"kubernetes.io/projected/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-kube-api-access-t2gkr\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:47.465098 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.465105    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:47.631998 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.631919    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" containerID="05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23" exitCode=0
Sep 10 15:12:47.631998 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.631977    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"
Sep 10 15:12:47.632210 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.632006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq" event={"ID":"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0","Type":"ContainerDied","Data":"05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23"}
Sep 10 15:12:47.632210 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.632056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq" event={"ID":"b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0","Type":"ContainerDied","Data":"1b93783dc81054a6f214f8ce7c162908ed3efab8fbac242ea939b345e7ec2c4f"}
Sep 10 15:12:47.632210 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.632074    2565 scope.go:117] "RemoveContainer" containerID="05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23"
Sep 10 15:12:47.641192 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.641178    2565 scope.go:117] "RemoveContainer" containerID="05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23"
Sep 10 15:12:47.641427 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:47.641409    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23\": container with ID starting with 05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23 not found: ID does not exist" containerID="05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23"
Sep 10 15:12:47.641466 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.641435    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23"} err="failed to get container status \"05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23\": rpc error: code = NotFound desc = could not find container \"05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23\": container with ID starting with 05e53c1acc43b1164e7a4fc2f9bbf0799c201eccd811cd89a0bbc7559aecee23 not found: ID does not exist"
Sep 10 15:12:47.653957 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.653935    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"]
Sep 10 15:12:47.657535 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.657515    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/sub-test-model-decc52-kserve-77ff8fb587-mh6qq"]
Sep 10 15:12:47.976737 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:47.976661    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" path="/var/lib/kubelet/pods/b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0/volumes"
Sep 10 15:12:48.069501 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.069468    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r" podUID="098cd6f6-04c6-403f-94ec-901d5b5bbb13" containerName="main" containerID="cri-o://617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a" gracePeriod=45
Sep 10 15:12:48.314131 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.314108    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:12:48.373964 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.373933    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj6qf\" (UniqueName: \"kubernetes.io/projected/098cd6f6-04c6-403f-94ec-901d5b5bbb13-kube-api-access-hj6qf\") pod \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") "
Sep 10 15:12:48.374146 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.373981    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tmp-dir\") pod \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") "
Sep 10 15:12:48.374146 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374077    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-dshm\") pod \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") "
Sep 10 15:12:48.374146 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-home\") pod \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") "
Sep 10 15:12:48.374323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374198    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tls-certs\") pod \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") "
Sep 10 15:12:48.374323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-model-cache\") pod \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\" (UID: \"098cd6f6-04c6-403f-94ec-901d5b5bbb13\") "
Sep 10 15:12:48.374323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "098cd6f6-04c6-403f-94ec-901d5b5bbb13" (UID: "098cd6f6-04c6-403f-94ec-901d5b5bbb13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:48.374482 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374396    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-home" (OuterVolumeSpecName: "home") pod "098cd6f6-04c6-403f-94ec-901d5b5bbb13" (UID: "098cd6f6-04c6-403f-94ec-901d5b5bbb13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:48.374545 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374525    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-model-cache" (OuterVolumeSpecName: "model-cache") pod "098cd6f6-04c6-403f-94ec-901d5b5bbb13" (UID: "098cd6f6-04c6-403f-94ec-901d5b5bbb13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:48.374613 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374595    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:48.374688 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.374618    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:48.376078 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.376059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098cd6f6-04c6-403f-94ec-901d5b5bbb13-kube-api-access-hj6qf" (OuterVolumeSpecName: "kube-api-access-hj6qf") pod "098cd6f6-04c6-403f-94ec-901d5b5bbb13" (UID: "098cd6f6-04c6-403f-94ec-901d5b5bbb13"). InnerVolumeSpecName "kube-api-access-hj6qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:12:48.376212 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.376194    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "098cd6f6-04c6-403f-94ec-901d5b5bbb13" (UID: "098cd6f6-04c6-403f-94ec-901d5b5bbb13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:12:48.376274 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.376241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-dshm" (OuterVolumeSpecName: "dshm") pod "098cd6f6-04c6-403f-94ec-901d5b5bbb13" (UID: "098cd6f6-04c6-403f-94ec-901d5b5bbb13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:12:48.475592 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.475567    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj6qf\" (UniqueName: \"kubernetes.io/projected/098cd6f6-04c6-403f-94ec-901d5b5bbb13-kube-api-access-hj6qf\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:48.475592 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.475589    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:48.475754 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.475599    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cd6f6-04c6-403f-94ec-901d5b5bbb13-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:48.475754 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.475608    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cd6f6-04c6-403f-94ec-901d5b5bbb13-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:48.639307 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.639278    2565 generic.go:356] "Generic (PLEG): container finished" podID="098cd6f6-04c6-403f-94ec-901d5b5bbb13" containerID="617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a" exitCode=0
Sep 10 15:12:48.639437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.639314    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"
Sep 10 15:12:48.639437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.639320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r" event={"ID":"098cd6f6-04c6-403f-94ec-901d5b5bbb13","Type":"ContainerDied","Data":"617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a"}
Sep 10 15:12:48.639437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.639347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r" event={"ID":"098cd6f6-04c6-403f-94ec-901d5b5bbb13","Type":"ContainerDied","Data":"2568611c7dce70dbdb9db96cf5ea4730c7e8286d73df324456a5ff9ec0722b0b"}
Sep 10 15:12:48.639437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.639367    2565 scope.go:117] "RemoveContainer" containerID="617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a"
Sep 10 15:12:48.648826 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.648808    2565 scope.go:117] "RemoveContainer" containerID="617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a"
Sep 10 15:12:48.649076 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:48.649056    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a\": container with ID starting with 617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a not found: ID does not exist" containerID="617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a"
Sep 10 15:12:48.649136 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.649090    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a"} err="failed to get container status \"617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a\": rpc error: code = NotFound desc = could not find container \"617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a\": container with ID starting with 617671280c4bbc330b14ac7b24633531f93cbdec6eb243b7c1b42b58504b432a not found: ID does not exist"
Sep 10 15:12:48.663565 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.663539    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"]
Sep 10 15:12:48.667119 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:48.667098    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/sub-test-model-d5a2df-kserve-96dbc587-gtn6r"]
Sep 10 15:12:49.976614 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:49.976579    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098cd6f6-04c6-403f-94ec-901d5b5bbb13" path="/var/lib/kubelet/pods/098cd6f6-04c6-403f-94ec-901d5b5bbb13/volumes"
Sep 10 15:12:50.877474 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:50.877444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:50.878298 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:50.877491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:50.889884 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:50.889861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:51.667023 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:51.666991    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:12:54.751956 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:54.751919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:12:57.627926 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.627891    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"]
Sep 10 15:12:57.628398 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.628151    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk" podUID="5cca069e-e2be-4cf4-b99e-546c322f0c7d" containerName="maas-api" containerID="cri-o://f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6" gracePeriod=30
Sep 10 15:12:57.867775 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.867749    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:12:57.964633 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.964549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-maas-api-tls\") pod \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") "
Sep 10 15:12:57.964633 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.964591    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca069e-e2be-4cf4-b99e-546c322f0c7d-openshift-service-ca\") pod \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") "
Sep 10 15:12:57.964886 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.964664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tztzz\" (UniqueName: \"kubernetes.io/projected/5cca069e-e2be-4cf4-b99e-546c322f0c7d-kube-api-access-tztzz\") pod \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") "
Sep 10 15:12:57.964886 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.964720    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-metrics-certs\") pod \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\" (UID: \"5cca069e-e2be-4cf4-b99e-546c322f0c7d\") "
Sep 10 15:12:57.965006 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.964969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cca069e-e2be-4cf4-b99e-546c322f0c7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5cca069e-e2be-4cf4-b99e-546c322f0c7d" (UID: "5cca069e-e2be-4cf4-b99e-546c322f0c7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:12:57.966835 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.966805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5cca069e-e2be-4cf4-b99e-546c322f0c7d" (UID: "5cca069e-e2be-4cf4-b99e-546c322f0c7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:12:57.966971 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.966834    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5cca069e-e2be-4cf4-b99e-546c322f0c7d" (UID: "5cca069e-e2be-4cf4-b99e-546c322f0c7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:12:57.966971 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:57.966852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cca069e-e2be-4cf4-b99e-546c322f0c7d-kube-api-access-tztzz" (OuterVolumeSpecName: "kube-api-access-tztzz") pod "5cca069e-e2be-4cf4-b99e-546c322f0c7d" (UID: "5cca069e-e2be-4cf4-b99e-546c322f0c7d"). InnerVolumeSpecName "kube-api-access-tztzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:12:58.065598 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.065560    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:58.065598 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.065587    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cca069e-e2be-4cf4-b99e-546c322f0c7d-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:58.065598 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.065596    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca069e-e2be-4cf4-b99e-546c322f0c7d-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:58.065873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.065609    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tztzz\" (UniqueName: \"kubernetes.io/projected/5cca069e-e2be-4cf4-b99e-546c322f0c7d-kube-api-access-tztzz\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:12:58.693782 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.693742    2565 generic.go:356] "Generic (PLEG): container finished" podID="5cca069e-e2be-4cf4-b99e-546c322f0c7d" containerID="f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6" exitCode=0
Sep 10 15:12:58.694261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.693798    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"
Sep 10 15:12:58.694261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.693800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk" event={"ID":"5cca069e-e2be-4cf4-b99e-546c322f0c7d","Type":"ContainerDied","Data":"f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6"}
Sep 10 15:12:58.694261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.693922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk" event={"ID":"5cca069e-e2be-4cf4-b99e-546c322f0c7d","Type":"ContainerDied","Data":"af35c0789b3e993234c64dd36754cfac71b7aa96430b18d1a5a47e7ff61d24fe"}
Sep 10 15:12:58.694261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.693946    2565 scope.go:117] "RemoveContainer" containerID="f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6"
Sep 10 15:12:58.702790 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.702768    2565 scope.go:117] "RemoveContainer" containerID="f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6"
Sep 10 15:12:58.703057 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:12:58.703038    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6\": container with ID starting with f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6 not found: ID does not exist" containerID="f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6"
Sep 10 15:12:58.703108 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.703067    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6"} err="failed to get container status \"f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6\": rpc error: code = NotFound desc = could not find container \"f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6\": container with ID starting with f5b5d5ed868fecd5f6b4768137963ee537967dac579118fed3e51f090142b2f6 not found: ID does not exist"
Sep 10 15:12:58.716967 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.716938    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"]
Sep 10 15:12:58.722264 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:58.722239    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d7aa05-5bb79f8c8d-gwgjk"]
Sep 10 15:12:59.977924 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:12:59.977892    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cca069e-e2be-4cf4-b99e-546c322f0c7d" path="/var/lib/kubelet/pods/5cca069e-e2be-4cf4-b99e-546c322f0c7d/volumes"
Sep 10 15:13:10.260221 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:10.260144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:13:16.541250 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541213    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"]
Sep 10 15:13:16.541660 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541604    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" containerName="main"
Sep 10 15:13:16.541660 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541615    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" containerName="main"
Sep 10 15:13:16.541660 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541623    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098cd6f6-04c6-403f-94ec-901d5b5bbb13" containerName="main"
Sep 10 15:13:16.541660 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541630    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="098cd6f6-04c6-403f-94ec-901d5b5bbb13" containerName="main"
Sep 10 15:13:16.541660 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541637    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cca069e-e2be-4cf4-b99e-546c322f0c7d" containerName="maas-api"
Sep 10 15:13:16.541660 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541656    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cca069e-e2be-4cf4-b99e-546c322f0c7d" containerName="maas-api"
Sep 10 15:13:16.541863 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541710    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ba0a12-c2a3-467e-b1fc-d8e9600bc5c0" containerName="main"
Sep 10 15:13:16.541863 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541718    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="098cd6f6-04c6-403f-94ec-901d5b5bbb13" containerName="main"
Sep 10 15:13:16.541863 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.541724    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cca069e-e2be-4cf4-b99e-546c322f0c7d" containerName="maas-api"
Sep 10 15:13:16.546320 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.546293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.549637 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.549613    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"kube-root-ca.crt\""
Sep 10 15:13:16.549637 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.549623    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"default-dockercfg-wfhrm\""
Sep 10 15:13:16.549821 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.549636    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"rate-test-model-d5a2df-kserve-self-signed-certs\""
Sep 10 15:13:16.549821 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.549623    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d5a2df\"/\"openshift-service-ca.crt\""
Sep 10 15:13:16.555555 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.555499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"]
Sep 10 15:13:16.644692 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.644634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1291d5de-6bff-4945-9ba2-4f6c917766cd-tls-certs\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.644873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.644726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrfm\" (UniqueName: \"kubernetes.io/projected/1291d5de-6bff-4945-9ba2-4f6c917766cd-kube-api-access-lsrfm\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.644873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.644768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-home\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.644873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.644866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-tmp-dir\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.644998 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.644889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-model-cache\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.644998 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.644987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-dshm\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746184 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-tmp-dir\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746376 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-model-cache\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746376 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-dshm\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746376 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1291d5de-6bff-4945-9ba2-4f6c917766cd-tls-certs\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746376 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrfm\" (UniqueName: \"kubernetes.io/projected/1291d5de-6bff-4945-9ba2-4f6c917766cd-kube-api-access-lsrfm\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746376 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-home\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746668 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-tmp-dir\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746740 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-home\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.746800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.746736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-model-cache\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.748471 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.748448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-dshm\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.748765 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.748747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1291d5de-6bff-4945-9ba2-4f6c917766cd-tls-certs\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.754844 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.754819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrfm\" (UniqueName: \"kubernetes.io/projected/1291d5de-6bff-4945-9ba2-4f6c917766cd-kube-api-access-lsrfm\") pod \"rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") " pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:16.856751 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:16.856716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:17.133226 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:17.132839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"]
Sep 10 15:13:17.798840 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:17.798801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq" event={"ID":"1291d5de-6bff-4945-9ba2-4f6c917766cd","Type":"ContainerStarted","Data":"4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2"}
Sep 10 15:13:17.799240 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:17.798858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq" event={"ID":"1291d5de-6bff-4945-9ba2-4f6c917766cd","Type":"ContainerStarted","Data":"97833347db2733a47f00ec30060ea4269b711f49f9f6d93fef091234d199bf17"}
Sep 10 15:13:17.815058 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:17.815006    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq" podStartSLOduration=1.81498779 podStartE2EDuration="1.81498779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:13:17.813875769 +0000 UTC m=+1532.383415027" watchObservedRunningTime="2026-09-10 15:13:17.81498779 +0000 UTC m=+1532.384527071"
Sep 10 15:13:20.448460 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:20.448425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:13:26.857060 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:26.857027    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:26.857060 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:26.857070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:26.869728 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:26.869696    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:27.863436 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:27.863411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:13:32.786141 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:32.786105    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"]
Sep 10 15:13:32.786576 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:32.786310    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb" podUID="32a2d075-98b6-4155-a704-b75646075be7" containerName="maas-api" containerID="cri-o://c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e" gracePeriod=30
Sep 10 15:13:33.031034 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.031012    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:13:33.098399 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.098367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls\") pod \"32a2d075-98b6-4155-a704-b75646075be7\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") "
Sep 10 15:13:33.098399 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.098398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-metrics-certs\") pod \"32a2d075-98b6-4155-a704-b75646075be7\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") "
Sep 10 15:13:33.098587 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.098432    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a2d075-98b6-4155-a704-b75646075be7-openshift-service-ca\") pod \"32a2d075-98b6-4155-a704-b75646075be7\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") "
Sep 10 15:13:33.098587 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.098575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmlpp\" (UniqueName: \"kubernetes.io/projected/32a2d075-98b6-4155-a704-b75646075be7-kube-api-access-vmlpp\") pod \"32a2d075-98b6-4155-a704-b75646075be7\" (UID: \"32a2d075-98b6-4155-a704-b75646075be7\") "
Sep 10 15:13:33.098868 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.098840    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a2d075-98b6-4155-a704-b75646075be7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32a2d075-98b6-4155-a704-b75646075be7" (UID: "32a2d075-98b6-4155-a704-b75646075be7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:13:33.099003 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.098938    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a2d075-98b6-4155-a704-b75646075be7-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:13:33.100655 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.100614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a2d075-98b6-4155-a704-b75646075be7-kube-api-access-vmlpp" (OuterVolumeSpecName: "kube-api-access-vmlpp") pod "32a2d075-98b6-4155-a704-b75646075be7" (UID: "32a2d075-98b6-4155-a704-b75646075be7"). InnerVolumeSpecName "kube-api-access-vmlpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:13:33.101052 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.101032    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32a2d075-98b6-4155-a704-b75646075be7" (UID: "32a2d075-98b6-4155-a704-b75646075be7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:13:33.101052 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.101043    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32a2d075-98b6-4155-a704-b75646075be7" (UID: "32a2d075-98b6-4155-a704-b75646075be7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:13:33.200249 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.200218    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:13:33.200249 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.200246    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a2d075-98b6-4155-a704-b75646075be7-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:13:33.200249 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.200255    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmlpp\" (UniqueName: \"kubernetes.io/projected/32a2d075-98b6-4155-a704-b75646075be7-kube-api-access-vmlpp\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:13:33.886552 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.886520    2565 generic.go:356] "Generic (PLEG): container finished" podID="32a2d075-98b6-4155-a704-b75646075be7" containerID="c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e" exitCode=0
Sep 10 15:13:33.886552 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.886556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb" event={"ID":"32a2d075-98b6-4155-a704-b75646075be7","Type":"ContainerDied","Data":"c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e"}
Sep 10 15:13:33.887041 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.886562    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"
Sep 10 15:13:33.887041 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.886598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb" event={"ID":"32a2d075-98b6-4155-a704-b75646075be7","Type":"ContainerDied","Data":"3cd8829269d8e69d00b895455be72d1b8f205d980246649f10876e58eb7daa4d"}
Sep 10 15:13:33.887041 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.886620    2565 scope.go:117] "RemoveContainer" containerID="c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e"
Sep 10 15:13:33.900965 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.900941    2565 scope.go:117] "RemoveContainer" containerID="c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e"
Sep 10 15:13:33.901261 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:13:33.901240    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e\": container with ID starting with c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e not found: ID does not exist" containerID="c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e"
Sep 10 15:13:33.901328 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.901273    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e"} err="failed to get container status \"c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e\": rpc error: code = NotFound desc = could not find container \"c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e\": container with ID starting with c9639d024bd4bd5d705de8acfaf3b42891de01367f756ef73c353aaed7716e3e not found: ID does not exist"
Sep 10 15:13:33.920762 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.920731    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"]
Sep 10 15:13:33.925432 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.925409    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1f985c-84d68f4f69-9nzjb"]
Sep 10 15:13:33.977590 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:33.977560    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a2d075-98b6-4155-a704-b75646075be7" path="/var/lib/kubelet/pods/32a2d075-98b6-4155-a704-b75646075be7/volumes"
Sep 10 15:13:35.758347 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:35.758315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:13:51.451580 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:51.451540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:13:57.452480 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:57.452437    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"]
Sep 10 15:13:58.543424 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:13:58.543382    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"]
Sep 10 15:14:02.756055 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:02.756018    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:14:12.473282 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.473244    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt" podUID="6b122d09-7226-403f-b3df-8e8944afb467" containerName="main" containerID="cri-o://0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52" gracePeriod=45
Sep 10 15:14:12.722882 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.722859    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:14:12.755793 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.755723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-tmp-dir\") pod \"6b122d09-7226-403f-b3df-8e8944afb467\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") "
Sep 10 15:14:12.755793 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.755785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-model-cache\") pod \"6b122d09-7226-403f-b3df-8e8944afb467\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") "
Sep 10 15:14:12.755946 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.755874    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-dshm\") pod \"6b122d09-7226-403f-b3df-8e8944afb467\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") "
Sep 10 15:14:12.755946 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.755929    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b122d09-7226-403f-b3df-8e8944afb467-tls-certs\") pod \"6b122d09-7226-403f-b3df-8e8944afb467\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") "
Sep 10 15:14:12.756012 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.755972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-home\") pod \"6b122d09-7226-403f-b3df-8e8944afb467\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") "
Sep 10 15:14:12.756012 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.755996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld9b8\" (UniqueName: \"kubernetes.io/projected/6b122d09-7226-403f-b3df-8e8944afb467-kube-api-access-ld9b8\") pod \"6b122d09-7226-403f-b3df-8e8944afb467\" (UID: \"6b122d09-7226-403f-b3df-8e8944afb467\") "
Sep 10 15:14:12.756012 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.755993    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b122d09-7226-403f-b3df-8e8944afb467" (UID: "6b122d09-7226-403f-b3df-8e8944afb467"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:12.756148 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.756027    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b122d09-7226-403f-b3df-8e8944afb467" (UID: "6b122d09-7226-403f-b3df-8e8944afb467"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:12.756202 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.756187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-home" (OuterVolumeSpecName: "home") pod "6b122d09-7226-403f-b3df-8e8944afb467" (UID: "6b122d09-7226-403f-b3df-8e8944afb467"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:12.756307 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.756292    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:12.756370 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.756307    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:12.756370 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.756316    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:12.758015 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.757991    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-dshm" (OuterVolumeSpecName: "dshm") pod "6b122d09-7226-403f-b3df-8e8944afb467" (UID: "6b122d09-7226-403f-b3df-8e8944afb467"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:12.758130 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.758068    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b122d09-7226-403f-b3df-8e8944afb467-kube-api-access-ld9b8" (OuterVolumeSpecName: "kube-api-access-ld9b8") pod "6b122d09-7226-403f-b3df-8e8944afb467" (UID: "6b122d09-7226-403f-b3df-8e8944afb467"). InnerVolumeSpecName "kube-api-access-ld9b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:14:12.758178 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.758136    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b122d09-7226-403f-b3df-8e8944afb467-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b122d09-7226-403f-b3df-8e8944afb467" (UID: "6b122d09-7226-403f-b3df-8e8944afb467"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:14:12.857222 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.857183    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9b8\" (UniqueName: \"kubernetes.io/projected/6b122d09-7226-403f-b3df-8e8944afb467-kube-api-access-ld9b8\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:12.857222 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.857214    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b122d09-7226-403f-b3df-8e8944afb467-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:12.857222 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:12.857223    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b122d09-7226-403f-b3df-8e8944afb467-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:13.098715 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.098685    2565 generic.go:356] "Generic (PLEG): container finished" podID="6b122d09-7226-403f-b3df-8e8944afb467" containerID="0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52" exitCode=0
Sep 10 15:14:13.098885 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.098725    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"
Sep 10 15:14:13.098885 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.098773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt" event={"ID":"6b122d09-7226-403f-b3df-8e8944afb467","Type":"ContainerDied","Data":"0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52"}
Sep 10 15:14:13.098885 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.098813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt" event={"ID":"6b122d09-7226-403f-b3df-8e8944afb467","Type":"ContainerDied","Data":"9dbb6b155b66abd09c8e0a4bc87a9ba972165017c27598a812bbf747f7023d9d"}
Sep 10 15:14:13.098885 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.098832    2565 scope.go:117] "RemoveContainer" containerID="0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52"
Sep 10 15:14:13.108567 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.108549    2565 scope.go:117] "RemoveContainer" containerID="0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52"
Sep 10 15:14:13.108917 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:14:13.108898    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52\": container with ID starting with 0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52 not found: ID does not exist" containerID="0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52"
Sep 10 15:14:13.108982 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.108928    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52"} err="failed to get container status \"0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52\": rpc error: code = NotFound desc = could not find container \"0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52\": container with ID starting with 0e8f45da04e0ce11563d1ce6d44e99b5a774765a7dd1506482017535fa5fbc52 not found: ID does not exist"
Sep 10 15:14:13.119669 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.119632    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"]
Sep 10 15:14:13.126203 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.126183    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-decc52/rate-test-model-decc52-kserve-7bdc95dc79-5d8dt"]
Sep 10 15:14:13.349167 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.349084    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:14:13.562541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.562497    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq" podUID="1291d5de-6bff-4945-9ba2-4f6c917766cd" containerName="main" containerID="cri-o://4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2" gracePeriod=45
Sep 10 15:14:13.796436 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.796411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:14:13.866266 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866229    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-home\") pod \"1291d5de-6bff-4945-9ba2-4f6c917766cd\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") "
Sep 10 15:14:13.866411 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866279    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-tmp-dir\") pod \"1291d5de-6bff-4945-9ba2-4f6c917766cd\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") "
Sep 10 15:14:13.866411 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsrfm\" (UniqueName: \"kubernetes.io/projected/1291d5de-6bff-4945-9ba2-4f6c917766cd-kube-api-access-lsrfm\") pod \"1291d5de-6bff-4945-9ba2-4f6c917766cd\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") "
Sep 10 15:14:13.866411 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1291d5de-6bff-4945-9ba2-4f6c917766cd-tls-certs\") pod \"1291d5de-6bff-4945-9ba2-4f6c917766cd\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") "
Sep 10 15:14:13.866564 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-model-cache\") pod \"1291d5de-6bff-4945-9ba2-4f6c917766cd\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") "
Sep 10 15:14:13.866564 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866459    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-dshm\") pod \"1291d5de-6bff-4945-9ba2-4f6c917766cd\" (UID: \"1291d5de-6bff-4945-9ba2-4f6c917766cd\") "
Sep 10 15:14:13.866564 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866533    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-home" (OuterVolumeSpecName: "home") pod "1291d5de-6bff-4945-9ba2-4f6c917766cd" (UID: "1291d5de-6bff-4945-9ba2-4f6c917766cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:13.866564 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866546    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1291d5de-6bff-4945-9ba2-4f6c917766cd" (UID: "1291d5de-6bff-4945-9ba2-4f6c917766cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:13.866800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866742    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "1291d5de-6bff-4945-9ba2-4f6c917766cd" (UID: "1291d5de-6bff-4945-9ba2-4f6c917766cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:13.866800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866757    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:13.866800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.866770    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:13.868384 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.868363    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1291d5de-6bff-4945-9ba2-4f6c917766cd-kube-api-access-lsrfm" (OuterVolumeSpecName: "kube-api-access-lsrfm") pod "1291d5de-6bff-4945-9ba2-4f6c917766cd" (UID: "1291d5de-6bff-4945-9ba2-4f6c917766cd"). InnerVolumeSpecName "kube-api-access-lsrfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:14:13.868459 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.868406    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-dshm" (OuterVolumeSpecName: "dshm") pod "1291d5de-6bff-4945-9ba2-4f6c917766cd" (UID: "1291d5de-6bff-4945-9ba2-4f6c917766cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:13.868542 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.868526    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1291d5de-6bff-4945-9ba2-4f6c917766cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1291d5de-6bff-4945-9ba2-4f6c917766cd" (UID: "1291d5de-6bff-4945-9ba2-4f6c917766cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:14:13.967541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.967449    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1291d5de-6bff-4945-9ba2-4f6c917766cd-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:13.967541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.967478    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:13.967541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.967487    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1291d5de-6bff-4945-9ba2-4f6c917766cd-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:13.967541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.967498    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsrfm\" (UniqueName: \"kubernetes.io/projected/1291d5de-6bff-4945-9ba2-4f6c917766cd-kube-api-access-lsrfm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:14:13.977764 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:13.977737    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b122d09-7226-403f-b3df-8e8944afb467" path="/var/lib/kubelet/pods/6b122d09-7226-403f-b3df-8e8944afb467/volumes"
Sep 10 15:14:14.105533 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.105500    2565 generic.go:356] "Generic (PLEG): container finished" podID="1291d5de-6bff-4945-9ba2-4f6c917766cd" containerID="4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2" exitCode=0
Sep 10 15:14:14.105736 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.105542    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"
Sep 10 15:14:14.105736 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.105592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq" event={"ID":"1291d5de-6bff-4945-9ba2-4f6c917766cd","Type":"ContainerDied","Data":"4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2"}
Sep 10 15:14:14.105736 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.105654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq" event={"ID":"1291d5de-6bff-4945-9ba2-4f6c917766cd","Type":"ContainerDied","Data":"97833347db2733a47f00ec30060ea4269b711f49f9f6d93fef091234d199bf17"}
Sep 10 15:14:14.105736 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.105675    2565 scope.go:117] "RemoveContainer" containerID="4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2"
Sep 10 15:14:14.114591 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.114559    2565 scope.go:117] "RemoveContainer" containerID="4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2"
Sep 10 15:14:14.114890 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:14:14.114861    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2\": container with ID starting with 4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2 not found: ID does not exist" containerID="4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2"
Sep 10 15:14:14.114962 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.114892    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2"} err="failed to get container status \"4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2\": rpc error: code = NotFound desc = could not find container \"4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2\": container with ID starting with 4946b9a9c1bffaeb7292939888e17d255bb022252bb98cabe31c3b18e3f495b2 not found: ID does not exist"
Sep 10 15:14:14.121842 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.121820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"]
Sep 10 15:14:14.125982 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:14.125961    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d5a2df/rate-test-model-d5a2df-kserve-6b4f768bf9-k25bq"]
Sep 10 15:14:15.976966 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:15.976934    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1291d5de-6bff-4945-9ba2-4f6c917766cd" path="/var/lib/kubelet/pods/1291d5de-6bff-4945-9ba2-4f6c917766cd/volumes"
Sep 10 15:14:33.160149 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160115    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"]
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160503    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1291d5de-6bff-4945-9ba2-4f6c917766cd" containerName="main"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160515    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1291d5de-6bff-4945-9ba2-4f6c917766cd" containerName="main"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160527    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a2d075-98b6-4155-a704-b75646075be7" containerName="maas-api"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160532    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a2d075-98b6-4155-a704-b75646075be7" containerName="maas-api"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160538    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b122d09-7226-403f-b3df-8e8944afb467" containerName="main"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160545    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b122d09-7226-403f-b3df-8e8944afb467" containerName="main"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160606    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a2d075-98b6-4155-a704-b75646075be7" containerName="maas-api"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160615    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1291d5de-6bff-4945-9ba2-4f6c917766cd" containerName="main"
Sep 10 15:14:33.160772 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.160623    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b122d09-7226-403f-b3df-8e8944afb467" containerName="main"
Sep 10 15:14:33.163896 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.163875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.166263 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.166238    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6333de-serving-cert\""
Sep 10 15:14:33.176522 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.176498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"]
Sep 10 15:14:33.243133 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.243100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-metrics-certs\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.243303 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.243145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eacc24c-e5e8-4214-af44-f87a34452919-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.243303 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.243171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvt9b\" (UniqueName: \"kubernetes.io/projected/6eacc24c-e5e8-4214-af44-f87a34452919-kube-api-access-cvt9b\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.243303 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.243238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.343787 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.343749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-metrics-certs\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.343987 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.343801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eacc24c-e5e8-4214-af44-f87a34452919-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.343987 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.343829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvt9b\" (UniqueName: \"kubernetes.io/projected/6eacc24c-e5e8-4214-af44-f87a34452919-kube-api-access-cvt9b\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.343987 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.343877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.344428 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.344402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eacc24c-e5e8-4214-af44-f87a34452919-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.346347 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.346329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-metrics-certs\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.346468 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.346447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.354871 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.354850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvt9b\" (UniqueName: \"kubernetes.io/projected/6eacc24c-e5e8-4214-af44-f87a34452919-kube-api-access-cvt9b\") pod \"maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.474184 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.474099    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:33.753793 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:33.753763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"]
Sep 10 15:14:33.754412 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:14:33.754391    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eacc24c_e5e8_4214_af44_f87a34452919.slice/crio-24ce942cbb98c2a5b2062fb4beba2a3911df17c0ab871f51f3550a23fc6a65de WatchSource:0}: Error finding container 24ce942cbb98c2a5b2062fb4beba2a3911df17c0ab871f51f3550a23fc6a65de: Status 404 returned error can't find the container with id 24ce942cbb98c2a5b2062fb4beba2a3911df17c0ab871f51f3550a23fc6a65de
Sep 10 15:14:34.209835 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:34.209800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk" event={"ID":"6eacc24c-e5e8-4214-af44-f87a34452919","Type":"ContainerStarted","Data":"6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f"}
Sep 10 15:14:34.210297 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:34.209853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk" event={"ID":"6eacc24c-e5e8-4214-af44-f87a34452919","Type":"ContainerStarted","Data":"24ce942cbb98c2a5b2062fb4beba2a3911df17c0ab871f51f3550a23fc6a65de"}
Sep 10 15:14:34.210297 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:34.209885    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:14:34.228167 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:34.228120    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk" podStartSLOduration=0.858150188 podStartE2EDuration="1.228106815s" totalImagesPullingTime="369.956627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:14:34.225684605 +0000 UTC m=+1608.795223863" watchObservedRunningTime="2026-09-10 15:14:34.228106815 +0000 UTC m=+1608.797646073"
Sep 10 15:14:40.053892 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:40.053810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:14:40.220278 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:14:40.220247    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:15:00.142297 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.142260    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"]
Sep 10 15:15:00.145720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.145702    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv"]
Sep 10 15:15:00.145830 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.145810    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"
Sep 10 15:15:00.148059 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.148039    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\""
Sep 10 15:15:00.148762 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.148746    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv"
Sep 10 15:15:00.154520 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.154158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv"]
Sep 10 15:15:00.156894 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.156867    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"]
Sep 10 15:15:00.158748 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.158724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"
Sep 10 15:15:00.166290 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.166273    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv"
Sep 10 15:15:00.439154 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.439093    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"]
Sep 10 15:15:00.441634 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:15:00.441610    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22bf12b_295a_4c28_82b8_e4f5c0f01d9c.slice/crio-90cc7b5e1de32032b922f972c7aa28e0757b420439ce02089c6c88fece2fa45b WatchSource:0}: Error finding container 90cc7b5e1de32032b922f972c7aa28e0757b420439ce02089c6c88fece2fa45b: Status 404 returned error can't find the container with id 90cc7b5e1de32032b922f972c7aa28e0757b420439ce02089c6c88fece2fa45b
Sep 10 15:15:00.458074 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:00.458048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv"]
Sep 10 15:15:00.459739 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:15:00.459710    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9265e256_20cb_43f6_9774_b6d7b22be0bf.slice/crio-e1f7d73fa09207f351c5181b037185c92b711132353c7206cf824deb74bef1c1 WatchSource:0}: Error finding container e1f7d73fa09207f351c5181b037185c92b711132353c7206cf824deb74bef1c1: Status 404 returned error can't find the container with id e1f7d73fa09207f351c5181b037185c92b711132353c7206cf824deb74bef1c1
Sep 10 15:15:01.355613 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:01.355577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv" event={"ID":"9265e256-20cb-43f6-9774-b6d7b22be0bf","Type":"ContainerStarted","Data":"e1f7d73fa09207f351c5181b037185c92b711132353c7206cf824deb74bef1c1"}
Sep 10 15:15:01.356809 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:01.356781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk" event={"ID":"f22bf12b-295a-4c28-82b8-e4f5c0f01d9c","Type":"ContainerStarted","Data":"90cc7b5e1de32032b922f972c7aa28e0757b420439ce02089c6c88fece2fa45b"}
Sep 10 15:15:03.369676 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:03.369542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk" event={"ID":"f22bf12b-295a-4c28-82b8-e4f5c0f01d9c","Type":"ContainerStarted","Data":"68e418a4b9be5e752b0977e72b5b99a9d7ec5b4b9dfe441e8c233559402b94c3"}
Sep 10 15:15:03.371165 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:03.371137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv" event={"ID":"9265e256-20cb-43f6-9774-b6d7b22be0bf","Type":"ContainerStarted","Data":"97c75769b9c6ca729ede63b6c1174182b8b952330a9f0a9181818723f0a23f48"}
Sep 10 15:15:03.384221 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:03.384155    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk" podStartSLOduration=0.999082252 podStartE2EDuration="3.384139061s" totalImagesPullingTime="2.385056809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:03.383390582 +0000 UTC m=+1637.952929841" watchObservedRunningTime="2026-09-10 15:15:03.384139061 +0000 UTC m=+1637.953678319"
Sep 10 15:15:03.396782 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:03.396724    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv" podStartSLOduration=1.025410563 podStartE2EDuration="3.396703967s" totalImagesPullingTime="2.371293404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:03.396424211 +0000 UTC m=+1637.965963481" watchObservedRunningTime="2026-09-10 15:15:03.396703967 +0000 UTC m=+1637.966243225"
Sep 10 15:15:04.253600 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.253564    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"]
Sep 10 15:15:04.257364 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.257346    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.259909 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.259890    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-13ee06\"/\"openshift-service-ca.crt\""
Sep 10 15:15:04.260752 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.260734    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-13ee06\"/\"kube-root-ca.crt\""
Sep 10 15:15:04.260847 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.260804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-13ee06\"/\"ipp-route-13ee06-kserve-self-signed-certs\""
Sep 10 15:15:04.260917 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.260862    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-13ee06\"/\"default-dockercfg-45hfv\""
Sep 10 15:15:04.265961 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.265940    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"]
Sep 10 15:15:04.320979 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.320941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tnv\" (UniqueName: \"kubernetes.io/projected/a627e29c-1144-49f2-85d4-dbd0a3739cc1-kube-api-access-w6tnv\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.320979 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.320986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tmp-dir\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.321272 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.321008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tls-certs\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.321272 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.321067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-home\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.321272 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.321105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-model-cache\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.321272 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.321164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-dshm\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.378299 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.378269    2565 generic.go:356] "Generic (PLEG): container finished" podID="9265e256-20cb-43f6-9774-b6d7b22be0bf" containerID="97c75769b9c6ca729ede63b6c1174182b8b952330a9f0a9181818723f0a23f48" exitCode=0
Sep 10 15:15:04.378778 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.378356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv" event={"ID":"9265e256-20cb-43f6-9774-b6d7b22be0bf","Type":"ContainerDied","Data":"97c75769b9c6ca729ede63b6c1174182b8b952330a9f0a9181818723f0a23f48"}
Sep 10 15:15:04.379983 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.379957    2565 generic.go:356] "Generic (PLEG): container finished" podID="f22bf12b-295a-4c28-82b8-e4f5c0f01d9c" containerID="68e418a4b9be5e752b0977e72b5b99a9d7ec5b4b9dfe441e8c233559402b94c3" exitCode=0
Sep 10 15:15:04.380115 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.380036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk" event={"ID":"f22bf12b-295a-4c28-82b8-e4f5c0f01d9c","Type":"ContainerDied","Data":"68e418a4b9be5e752b0977e72b5b99a9d7ec5b4b9dfe441e8c233559402b94c3"}
Sep 10 15:15:04.422316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tls-certs\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422316 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-home\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422545 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-model-cache\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422545 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-dshm\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422545 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tnv\" (UniqueName: \"kubernetes.io/projected/a627e29c-1144-49f2-85d4-dbd0a3739cc1-kube-api-access-w6tnv\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422545 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tmp-dir\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422823 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-home\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422923 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-model-cache\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.422923 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.422899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tmp-dir\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.424704 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.424683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-dshm\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.424924 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.424906    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tls-certs\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.432651 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.432608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tnv\" (UniqueName: \"kubernetes.io/projected/a627e29c-1144-49f2-85d4-dbd0a3739cc1-kube-api-access-w6tnv\") pod \"ipp-route-13ee06-kserve-6744bb7cf5-wtzmg\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") " pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.568018 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.567925    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:04.841002 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:04.840975    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"]
Sep 10 15:15:04.841663 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:15:04.841622    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda627e29c_1144_49f2_85d4_dbd0a3739cc1.slice/crio-4af5ef5c46c18773a8e5939211606c4835b5374dc206c8b1a3726c2a5ea71d7c WatchSource:0}: Error finding container 4af5ef5c46c18773a8e5939211606c4835b5374dc206c8b1a3726c2a5ea71d7c: Status 404 returned error can't find the container with id 4af5ef5c46c18773a8e5939211606c4835b5374dc206c8b1a3726c2a5ea71d7c
Sep 10 15:15:05.387187 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:05.387146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg" event={"ID":"a627e29c-1144-49f2-85d4-dbd0a3739cc1","Type":"ContainerStarted","Data":"69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83"}
Sep 10 15:15:05.387616 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:05.387197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg" event={"ID":"a627e29c-1144-49f2-85d4-dbd0a3739cc1","Type":"ContainerStarted","Data":"4af5ef5c46c18773a8e5939211606c4835b5374dc206c8b1a3726c2a5ea71d7c"}
Sep 10 15:15:05.404455 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:05.404399    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg" podStartSLOduration=1.404380058 podStartE2EDuration="1.404380058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:05.403148548 +0000 UTC m=+1639.972687810" watchObservedRunningTime="2026-09-10 15:15:05.404380058 +0000 UTC m=+1639.973919366"
Sep 10 15:15:05.528282 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:05.528261    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv"
Sep 10 15:15:05.544606 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:05.544577    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"
Sep 10 15:15:06.394495 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:06.394467    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv"
Sep 10 15:15:06.395003 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:06.394472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817555-h8hsv" event={"ID":"9265e256-20cb-43f6-9774-b6d7b22be0bf","Type":"ContainerDied","Data":"e1f7d73fa09207f351c5181b037185c92b711132353c7206cf824deb74bef1c1"}
Sep 10 15:15:06.395003 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:06.394602    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1f7d73fa09207f351c5181b037185c92b711132353c7206cf824deb74bef1c1"
Sep 10 15:15:06.395845 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:06.395828    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"
Sep 10 15:15:06.395959 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:06.395878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk" event={"ID":"f22bf12b-295a-4c28-82b8-e4f5c0f01d9c","Type":"ContainerDied","Data":"90cc7b5e1de32032b922f972c7aa28e0757b420439ce02089c6c88fece2fa45b"}
Sep 10 15:15:06.395959 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:06.395926    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90cc7b5e1de32032b922f972c7aa28e0757b420439ce02089c6c88fece2fa45b"
Sep 10 15:15:08.257726 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:08.257685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:15:14.568690 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:14.568650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:14.569161 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:14.568709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:14.581870 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:14.581842    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:15.460870 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:15.460837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:15:36.253637 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:36.253602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:15:47.397829 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.397795    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"]
Sep 10 15:15:47.398204 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.398166    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22bf12b-295a-4c28-82b8-e4f5c0f01d9c" containerName="cleanup"
Sep 10 15:15:47.398204 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.398177    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22bf12b-295a-4c28-82b8-e4f5c0f01d9c" containerName="cleanup"
Sep 10 15:15:47.398204 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.398188    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9265e256-20cb-43f6-9774-b6d7b22be0bf" containerName="cleanup"
Sep 10 15:15:47.398204 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.398194    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9265e256-20cb-43f6-9774-b6d7b22be0bf" containerName="cleanup"
Sep 10 15:15:47.398340 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.398251    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9265e256-20cb-43f6-9774-b6d7b22be0bf" containerName="cleanup"
Sep 10 15:15:47.398340 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.398262    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22bf12b-295a-4c28-82b8-e4f5c0f01d9c" containerName="cleanup"
Sep 10 15:15:47.405865 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.405846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.408114 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.408091    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-06cc0e-serving-cert\""
Sep 10 15:15:47.412120 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.412098    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"]
Sep 10 15:15:47.419298 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.419278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfs8m\" (UniqueName: \"kubernetes.io/projected/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-kube-api-access-zfs8m\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.419403 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.419318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.419403 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.419339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.419517 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.419426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.520190 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.520152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.520190 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.520191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.520447 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.520216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.520447 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.520327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfs8m\" (UniqueName: \"kubernetes.io/projected/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-kube-api-access-zfs8m\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.520447 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:15:47.520366    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-serving-cert: secret "maas-api-e2e-ipp-cleanup-06cc0e-serving-cert" not found
Sep 10 15:15:47.520447 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:15:47.520446    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls podName:b8206ee6-9dbd-4149-bf8b-e5e7e481c301 nodeName:}" failed. No retries permitted until 2026-09-10 15:15:48.020422838 +0000 UTC m=+1682.589962091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh" (UID: "b8206ee6-9dbd-4149-bf8b-e5e7e481c301") : secret "maas-api-e2e-ipp-cleanup-06cc0e-serving-cert" not found
Sep 10 15:15:47.520870 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.520851    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.522808 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.522791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:47.530778 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:47.530756    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfs8m\" (UniqueName: \"kubernetes.io/projected/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-kube-api-access-zfs8m\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:48.024409 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:48.024374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:48.026892 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:48.026866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:48.317957 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:48.317879    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:48.589515 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:48.589482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"]
Sep 10 15:15:48.590983 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:15:48.590958    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8206ee6_9dbd_4149_bf8b_e5e7e481c301.slice/crio-d221a478d4498b96fc5f81e02bd693564658c26f66b00250c7124b67dfd3c063 WatchSource:0}: Error finding container d221a478d4498b96fc5f81e02bd693564658c26f66b00250c7124b67dfd3c063: Status 404 returned error can't find the container with id d221a478d4498b96fc5f81e02bd693564658c26f66b00250c7124b67dfd3c063
Sep 10 15:15:48.624164 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:48.624134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh" event={"ID":"b8206ee6-9dbd-4149-bf8b-e5e7e481c301","Type":"ContainerStarted","Data":"d221a478d4498b96fc5f81e02bd693564658c26f66b00250c7124b67dfd3c063"}
Sep 10 15:15:49.631308 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:49.631276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh" event={"ID":"b8206ee6-9dbd-4149-bf8b-e5e7e481c301","Type":"ContainerStarted","Data":"01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e"}
Sep 10 15:15:49.631777 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:49.631335    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:49.652310 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:49.652262    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh" podStartSLOduration=2.090485863 podStartE2EDuration="2.652246455s" totalImagesPullingTime="561.760592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:49.650286859 +0000 UTC m=+1684.219826116" watchObservedRunningTime="2026-09-10 15:15:49.652246455 +0000 UTC m=+1684.221785713"
Sep 10 15:15:55.643160 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:55.643130    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:15:57.255054 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:15:57.255023    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:16:17.999239 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:17.999149    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"]
Sep 10 15:16:17.999816 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:17.999395    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh" podUID="b8206ee6-9dbd-4149-bf8b-e5e7e481c301" containerName="maas-api" containerID="cri-o://01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e" gracePeriod=30
Sep 10 15:16:18.239838 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.239815    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:16:18.406127 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.406097    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-openshift-service-ca\") pod \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") "
Sep 10 15:16:18.406329 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.406177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-metrics-certs\") pod \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") "
Sep 10 15:16:18.406329 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.406214    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfs8m\" (UniqueName: \"kubernetes.io/projected/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-kube-api-access-zfs8m\") pod \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") "
Sep 10 15:16:18.406329 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.406254    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls\") pod \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\" (UID: \"b8206ee6-9dbd-4149-bf8b-e5e7e481c301\") "
Sep 10 15:16:18.406544 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.406518    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8206ee6-9dbd-4149-bf8b-e5e7e481c301" (UID: "b8206ee6-9dbd-4149-bf8b-e5e7e481c301"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:16:18.406667 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.406651    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:16:18.408486 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.408456    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-kube-api-access-zfs8m" (OuterVolumeSpecName: "kube-api-access-zfs8m") pod "b8206ee6-9dbd-4149-bf8b-e5e7e481c301" (UID: "b8206ee6-9dbd-4149-bf8b-e5e7e481c301"). InnerVolumeSpecName "kube-api-access-zfs8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:16:18.408819 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.408798    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b8206ee6-9dbd-4149-bf8b-e5e7e481c301" (UID: "b8206ee6-9dbd-4149-bf8b-e5e7e481c301"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:16:18.408903 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.408819    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8206ee6-9dbd-4149-bf8b-e5e7e481c301" (UID: "b8206ee6-9dbd-4149-bf8b-e5e7e481c301"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:16:18.507637 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.507604    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:16:18.507637 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.507632    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfs8m\" (UniqueName: \"kubernetes.io/projected/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-kube-api-access-zfs8m\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:16:18.507839 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.507665    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8206ee6-9dbd-4149-bf8b-e5e7e481c301-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:16:18.807198 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.807102    2565 generic.go:356] "Generic (PLEG): container finished" podID="b8206ee6-9dbd-4149-bf8b-e5e7e481c301" containerID="01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e" exitCode=0
Sep 10 15:16:18.807198 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.807150    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"
Sep 10 15:16:18.807448 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.807200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh" event={"ID":"b8206ee6-9dbd-4149-bf8b-e5e7e481c301","Type":"ContainerDied","Data":"01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e"}
Sep 10 15:16:18.807448 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.807256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh" event={"ID":"b8206ee6-9dbd-4149-bf8b-e5e7e481c301","Type":"ContainerDied","Data":"d221a478d4498b96fc5f81e02bd693564658c26f66b00250c7124b67dfd3c063"}
Sep 10 15:16:18.807448 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.807284    2565 scope.go:117] "RemoveContainer" containerID="01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e"
Sep 10 15:16:18.818756 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.818737    2565 scope.go:117] "RemoveContainer" containerID="01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e"
Sep 10 15:16:18.819409 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:16:18.819382    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e\": container with ID starting with 01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e not found: ID does not exist" containerID="01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e"
Sep 10 15:16:18.819512 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.819424    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e"} err="failed to get container status \"01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e\": rpc error: code = NotFound desc = could not find container \"01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e\": container with ID starting with 01318679cabd010746b32ab48aac524c6bfb7e885c5ddfcd1558e566cb2edb2e not found: ID does not exist"
Sep 10 15:16:18.833723 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.833700    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"]
Sep 10 15:16:18.837855 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:18.837834    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-06cc0e-b4b8b6476-g24vh"]
Sep 10 15:16:19.976547 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:19.976503    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8206ee6-9dbd-4149-bf8b-e5e7e481c301" path="/var/lib/kubelet/pods/b8206ee6-9dbd-4149-bf8b-e5e7e481c301/volumes"
Sep 10 15:16:22.647928 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:22.647890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:16:41.350217 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:41.350181    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:16:43.568436 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:43.568395    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"]
Sep 10 15:16:43.571134 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:43.571104    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-13ee06-29817555-fplkk"]
Sep 10 15:16:43.976502 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:43.976467    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22bf12b-295a-4c28-82b8-e4f5c0f01d9c" path="/var/lib/kubelet/pods/f22bf12b-295a-4c28-82b8-e4f5c0f01d9c/volumes"
Sep 10 15:16:49.108919 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:49.108881    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"]
Sep 10 15:16:54.370630 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:16:54.370588    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:17:04.128424 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.128372    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg" podUID="a627e29c-1144-49f2-85d4-dbd0a3739cc1" containerName="main" containerID="cri-o://69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83" gracePeriod=45
Sep 10 15:17:04.376741 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.376712    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:17:04.526238 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526140    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tmp-dir\") pod \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") "
Sep 10 15:17:04.526422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-home\") pod \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") "
Sep 10 15:17:04.526422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tls-certs\") pod \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") "
Sep 10 15:17:04.526422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526297    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-model-cache\") pod \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") "
Sep 10 15:17:04.526422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526313    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6tnv\" (UniqueName: \"kubernetes.io/projected/a627e29c-1144-49f2-85d4-dbd0a3739cc1-kube-api-access-w6tnv\") pod \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") "
Sep 10 15:17:04.526422 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-dshm\") pod \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\" (UID: \"a627e29c-1144-49f2-85d4-dbd0a3739cc1\") "
Sep 10 15:17:04.526728 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526425    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a627e29c-1144-49f2-85d4-dbd0a3739cc1" (UID: "a627e29c-1144-49f2-85d4-dbd0a3739cc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:17:04.526728 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526598    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-home" (OuterVolumeSpecName: "home") pod "a627e29c-1144-49f2-85d4-dbd0a3739cc1" (UID: "a627e29c-1144-49f2-85d4-dbd0a3739cc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:17:04.526728 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a627e29c-1144-49f2-85d4-dbd0a3739cc1" (UID: "a627e29c-1144-49f2-85d4-dbd0a3739cc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:17:04.526728 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526710    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:04.526873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.526734    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:04.528581 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.528550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-dshm" (OuterVolumeSpecName: "dshm") pod "a627e29c-1144-49f2-85d4-dbd0a3739cc1" (UID: "a627e29c-1144-49f2-85d4-dbd0a3739cc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:17:04.528705 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.528613    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a627e29c-1144-49f2-85d4-dbd0a3739cc1" (UID: "a627e29c-1144-49f2-85d4-dbd0a3739cc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:17:04.528705 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.528622    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a627e29c-1144-49f2-85d4-dbd0a3739cc1-kube-api-access-w6tnv" (OuterVolumeSpecName: "kube-api-access-w6tnv") pod "a627e29c-1144-49f2-85d4-dbd0a3739cc1" (UID: "a627e29c-1144-49f2-85d4-dbd0a3739cc1"). InnerVolumeSpecName "kube-api-access-w6tnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:17:04.628163 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.628119    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:04.628163 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.628156    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a627e29c-1144-49f2-85d4-dbd0a3739cc1-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:04.628163 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.628168    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a627e29c-1144-49f2-85d4-dbd0a3739cc1-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:04.628407 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:04.628181    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tnv\" (UniqueName: \"kubernetes.io/projected/a627e29c-1144-49f2-85d4-dbd0a3739cc1-kube-api-access-w6tnv\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:05.056216 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.056185    2565 generic.go:356] "Generic (PLEG): container finished" podID="a627e29c-1144-49f2-85d4-dbd0a3739cc1" containerID="69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83" exitCode=0
Sep 10 15:17:05.056437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.056235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg" event={"ID":"a627e29c-1144-49f2-85d4-dbd0a3739cc1","Type":"ContainerDied","Data":"69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83"}
Sep 10 15:17:05.056437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.056241    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"
Sep 10 15:17:05.056437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.056261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg" event={"ID":"a627e29c-1144-49f2-85d4-dbd0a3739cc1","Type":"ContainerDied","Data":"4af5ef5c46c18773a8e5939211606c4835b5374dc206c8b1a3726c2a5ea71d7c"}
Sep 10 15:17:05.056437 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.056277    2565 scope.go:117] "RemoveContainer" containerID="69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83"
Sep 10 15:17:05.065765 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.065748    2565 scope.go:117] "RemoveContainer" containerID="69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83"
Sep 10 15:17:05.066004 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:17:05.065985    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83\": container with ID starting with 69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83 not found: ID does not exist" containerID="69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83"
Sep 10 15:17:05.066069 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.066018    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83"} err="failed to get container status \"69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83\": rpc error: code = NotFound desc = could not find container \"69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83\": container with ID starting with 69feca2918a9c5b2cdf8af9bbbd4715682d21a1acb48bbf3772861e323b0fd83 not found: ID does not exist"
Sep 10 15:17:05.078319 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.078290    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"]
Sep 10 15:17:05.080501 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.080480    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-13ee06/ipp-route-13ee06-kserve-6744bb7cf5-wtzmg"]
Sep 10 15:17:05.976853 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:05.976810    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a627e29c-1144-49f2-85d4-dbd0a3739cc1" path="/var/lib/kubelet/pods/a627e29c-1144-49f2-85d4-dbd0a3739cc1/volumes"
Sep 10 15:17:42.944062 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:42.943976    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"]
Sep 10 15:17:42.944554 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:42.944168    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk" podUID="6eacc24c-e5e8-4214-af44-f87a34452919" containerName="maas-api" containerID="cri-o://6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f" gracePeriod=30
Sep 10 15:17:43.191418 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.191394    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:17:43.262116 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.262036    2565 generic.go:356] "Generic (PLEG): container finished" podID="6eacc24c-e5e8-4214-af44-f87a34452919" containerID="6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f" exitCode=0
Sep 10 15:17:43.262116 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.262078    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"
Sep 10 15:17:43.262287 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.262121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk" event={"ID":"6eacc24c-e5e8-4214-af44-f87a34452919","Type":"ContainerDied","Data":"6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f"}
Sep 10 15:17:43.262287 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.262170    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk" event={"ID":"6eacc24c-e5e8-4214-af44-f87a34452919","Type":"ContainerDied","Data":"24ce942cbb98c2a5b2062fb4beba2a3911df17c0ab871f51f3550a23fc6a65de"}
Sep 10 15:17:43.262287 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.262192    2565 scope.go:117] "RemoveContainer" containerID="6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f"
Sep 10 15:17:43.270742 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.270725    2565 scope.go:117] "RemoveContainer" containerID="6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f"
Sep 10 15:17:43.271018 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:17:43.270997    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f\": container with ID starting with 6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f not found: ID does not exist" containerID="6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f"
Sep 10 15:17:43.271083 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.271030    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f"} err="failed to get container status \"6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f\": rpc error: code = NotFound desc = could not find container \"6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f\": container with ID starting with 6f1dc80cd7f55de861a384c9af37dc209e355ef5990ee9a1050fac56b1f0da8f not found: ID does not exist"
Sep 10 15:17:43.279457 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.279439    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eacc24c-e5e8-4214-af44-f87a34452919-openshift-service-ca\") pod \"6eacc24c-e5e8-4214-af44-f87a34452919\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") "
Sep 10 15:17:43.279536 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.279483    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-maas-api-tls\") pod \"6eacc24c-e5e8-4214-af44-f87a34452919\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") "
Sep 10 15:17:43.279536 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.279527    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvt9b\" (UniqueName: \"kubernetes.io/projected/6eacc24c-e5e8-4214-af44-f87a34452919-kube-api-access-cvt9b\") pod \"6eacc24c-e5e8-4214-af44-f87a34452919\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") "
Sep 10 15:17:43.279608 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.279562    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-metrics-certs\") pod \"6eacc24c-e5e8-4214-af44-f87a34452919\" (UID: \"6eacc24c-e5e8-4214-af44-f87a34452919\") "
Sep 10 15:17:43.279864 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.279830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eacc24c-e5e8-4214-af44-f87a34452919-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6eacc24c-e5e8-4214-af44-f87a34452919" (UID: "6eacc24c-e5e8-4214-af44-f87a34452919"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:17:43.279992 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.279974    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eacc24c-e5e8-4214-af44-f87a34452919-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:43.281571 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.281548    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6eacc24c-e5e8-4214-af44-f87a34452919" (UID: "6eacc24c-e5e8-4214-af44-f87a34452919"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:17:43.281682 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.281587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eacc24c-e5e8-4214-af44-f87a34452919-kube-api-access-cvt9b" (OuterVolumeSpecName: "kube-api-access-cvt9b") pod "6eacc24c-e5e8-4214-af44-f87a34452919" (UID: "6eacc24c-e5e8-4214-af44-f87a34452919"). InnerVolumeSpecName "kube-api-access-cvt9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:17:43.281682 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.281614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6eacc24c-e5e8-4214-af44-f87a34452919" (UID: "6eacc24c-e5e8-4214-af44-f87a34452919"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:17:43.380517 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.380482    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:43.380517 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.380512    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvt9b\" (UniqueName: \"kubernetes.io/projected/6eacc24c-e5e8-4214-af44-f87a34452919-kube-api-access-cvt9b\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:43.380517 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.380522    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eacc24c-e5e8-4214-af44-f87a34452919-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:17:43.592156 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.592119    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"]
Sep 10 15:17:43.600909 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.600875    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6333de-565fb74657-c5xjk"]
Sep 10 15:17:43.978533 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:43.978493    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eacc24c-e5e8-4214-af44-f87a34452919" path="/var/lib/kubelet/pods/6eacc24c-e5e8-4214-af44-f87a34452919/volumes"
Sep 10 15:17:46.574161 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:17:46.574082    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:18:00.481275 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.481239    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"]
Sep 10 15:18:00.481833 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.481546    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h" podUID="95d9bc67-33d3-4f57-8f13-511a17d43468" containerName="maas-api" containerID="cri-o://832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805" gracePeriod=30
Sep 10 15:18:00.729811 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.729784    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:18:00.740607 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.740556    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d9bc67-33d3-4f57-8f13-511a17d43468-openshift-service-ca\") pod \"95d9bc67-33d3-4f57-8f13-511a17d43468\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") "
Sep 10 15:18:00.740607 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.740604    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6mrq\" (UniqueName: \"kubernetes.io/projected/95d9bc67-33d3-4f57-8f13-511a17d43468-kube-api-access-p6mrq\") pod \"95d9bc67-33d3-4f57-8f13-511a17d43468\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") "
Sep 10 15:18:00.740795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.740627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-metrics-certs\") pod \"95d9bc67-33d3-4f57-8f13-511a17d43468\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") "
Sep 10 15:18:00.740795 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.740717    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls\") pod \"95d9bc67-33d3-4f57-8f13-511a17d43468\" (UID: \"95d9bc67-33d3-4f57-8f13-511a17d43468\") "
Sep 10 15:18:00.740987 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.740959    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d9bc67-33d3-4f57-8f13-511a17d43468-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95d9bc67-33d3-4f57-8f13-511a17d43468" (UID: "95d9bc67-33d3-4f57-8f13-511a17d43468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:18:00.742873 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.742852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d9bc67-33d3-4f57-8f13-511a17d43468-kube-api-access-p6mrq" (OuterVolumeSpecName: "kube-api-access-p6mrq") pod "95d9bc67-33d3-4f57-8f13-511a17d43468" (UID: "95d9bc67-33d3-4f57-8f13-511a17d43468"). InnerVolumeSpecName "kube-api-access-p6mrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:18:00.742971 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.742880    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "95d9bc67-33d3-4f57-8f13-511a17d43468" (UID: "95d9bc67-33d3-4f57-8f13-511a17d43468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:18:00.742971 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.742891    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95d9bc67-33d3-4f57-8f13-511a17d43468" (UID: "95d9bc67-33d3-4f57-8f13-511a17d43468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:18:00.841720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.841686    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d9bc67-33d3-4f57-8f13-511a17d43468-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:18:00.841720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.841713    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6mrq\" (UniqueName: \"kubernetes.io/projected/95d9bc67-33d3-4f57-8f13-511a17d43468-kube-api-access-p6mrq\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:18:00.841720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.841723    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:18:00.841720 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:00.841732    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d9bc67-33d3-4f57-8f13-511a17d43468-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:18:01.359156 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.359122    2565 generic.go:356] "Generic (PLEG): container finished" podID="95d9bc67-33d3-4f57-8f13-511a17d43468" containerID="832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805" exitCode=0
Sep 10 15:18:01.359331 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.359194    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"
Sep 10 15:18:01.359331 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.359203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h" event={"ID":"95d9bc67-33d3-4f57-8f13-511a17d43468","Type":"ContainerDied","Data":"832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805"}
Sep 10 15:18:01.359331 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.359293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h" event={"ID":"95d9bc67-33d3-4f57-8f13-511a17d43468","Type":"ContainerDied","Data":"d4571b723d2545297a2dde1c6b52d79fe2493e174a909bb983790e58d557819a"}
Sep 10 15:18:01.359331 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.359323    2565 scope.go:117] "RemoveContainer" containerID="832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805"
Sep 10 15:18:01.368696 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.368676    2565 scope.go:117] "RemoveContainer" containerID="832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805"
Sep 10 15:18:01.368956 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:18:01.368938    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805\": container with ID starting with 832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805 not found: ID does not exist" containerID="832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805"
Sep 10 15:18:01.369007 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.368966    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805"} err="failed to get container status \"832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805\": rpc error: code = NotFound desc = could not find container \"832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805\": container with ID starting with 832ed2750cfd9d9b36b01be7fa2b79a724370c453ccf25741ed9604c0ef42805 not found: ID does not exist"
Sep 10 15:18:01.380087 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.380062    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"]
Sep 10 15:18:01.383962 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.383935    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-decc52-697556ddf7-svs2h"]
Sep 10 15:18:01.977069 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:01.977030    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d9bc67-33d3-4f57-8f13-511a17d43468" path="/var/lib/kubelet/pods/95d9bc67-33d3-4f57-8f13-511a17d43468/volumes"
Sep 10 15:18:03.349078 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:03.349041    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:18:07.551728 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.551690    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"]
Sep 10 15:18:07.552248 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552225    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627e29c-1144-49f2-85d4-dbd0a3739cc1" containerName="main"
Sep 10 15:18:07.552333 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552252    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627e29c-1144-49f2-85d4-dbd0a3739cc1" containerName="main"
Sep 10 15:18:07.552333 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552278    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d9bc67-33d3-4f57-8f13-511a17d43468" containerName="maas-api"
Sep 10 15:18:07.552333 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552287    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d9bc67-33d3-4f57-8f13-511a17d43468" containerName="maas-api"
Sep 10 15:18:07.552333 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552305    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8206ee6-9dbd-4149-bf8b-e5e7e481c301" containerName="maas-api"
Sep 10 15:18:07.552333 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552313    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8206ee6-9dbd-4149-bf8b-e5e7e481c301" containerName="maas-api"
Sep 10 15:18:07.552583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552336    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eacc24c-e5e8-4214-af44-f87a34452919" containerName="maas-api"
Sep 10 15:18:07.552583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552344    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eacc24c-e5e8-4214-af44-f87a34452919" containerName="maas-api"
Sep 10 15:18:07.552583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552423    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8206ee6-9dbd-4149-bf8b-e5e7e481c301" containerName="maas-api"
Sep 10 15:18:07.552583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552437    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eacc24c-e5e8-4214-af44-f87a34452919" containerName="maas-api"
Sep 10 15:18:07.552583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552447    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d9bc67-33d3-4f57-8f13-511a17d43468" containerName="maas-api"
Sep 10 15:18:07.552583 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.552458    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627e29c-1144-49f2-85d4-dbd0a3739cc1" containerName="main"
Sep 10 15:18:07.557021 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.556996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"
Sep 10 15:18:07.559624 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.559600    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:18:07.559775 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.559600    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhdvs\""
Sep 10 15:18:07.564108 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.564075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"]
Sep 10 15:18:07.603673 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.603602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2379ca3-a0a6-4771-a2b7-469180ba520f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt\" (UID: \"b2379ca3-a0a6-4771-a2b7-469180ba520f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"
Sep 10 15:18:07.705010 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.704972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2379ca3-a0a6-4771-a2b7-469180ba520f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt\" (UID: \"b2379ca3-a0a6-4771-a2b7-469180ba520f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"
Sep 10 15:18:07.705691 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.705631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2379ca3-a0a6-4771-a2b7-469180ba520f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt\" (UID: \"b2379ca3-a0a6-4771-a2b7-469180ba520f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"
Sep 10 15:18:07.868261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:07.868224    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"
Sep 10 15:18:08.139244 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:08.139004    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"]
Sep 10 15:18:08.142800 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:18:08.142773    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2379ca3_a0a6_4771_a2b7_469180ba520f.slice/crio-7414525e34434848378d1c83f14faea5716d67f7cb430392db91db7f91de40bd WatchSource:0}: Error finding container 7414525e34434848378d1c83f14faea5716d67f7cb430392db91db7f91de40bd: Status 404 returned error can't find the container with id 7414525e34434848378d1c83f14faea5716d67f7cb430392db91db7f91de40bd
Sep 10 15:18:08.397071 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:08.396979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt" event={"ID":"b2379ca3-a0a6-4771-a2b7-469180ba520f","Type":"ContainerStarted","Data":"11e9381b8ac19f3ca2e4c8b95366be543c35b48367b68bcb87c1edcce46457c4"}
Sep 10 15:18:08.397071 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:08.397038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt" event={"ID":"b2379ca3-a0a6-4771-a2b7-469180ba520f","Type":"ContainerStarted","Data":"7414525e34434848378d1c83f14faea5716d67f7cb430392db91db7f91de40bd"}
Sep 10 15:18:08.411814 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:08.411758    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt" podStartSLOduration=1.411742432 podStartE2EDuration="1.411742432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:18:08.410855225 +0000 UTC m=+1822.980394482" watchObservedRunningTime="2026-09-10 15:18:08.411742432 +0000 UTC m=+1822.981281690"
Sep 10 15:18:09.404474 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:09.404440    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2379ca3-a0a6-4771-a2b7-469180ba520f" containerID="11e9381b8ac19f3ca2e4c8b95366be543c35b48367b68bcb87c1edcce46457c4" exitCode=0
Sep 10 15:18:09.404887 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:09.404525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt" event={"ID":"b2379ca3-a0a6-4771-a2b7-469180ba520f","Type":"ContainerDied","Data":"11e9381b8ac19f3ca2e4c8b95366be543c35b48367b68bcb87c1edcce46457c4"}
Sep 10 15:18:10.536038 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:10.536018    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"
Sep 10 15:18:10.631625 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:10.631589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2379ca3-a0a6-4771-a2b7-469180ba520f-maas-api-service-ca\") pod \"b2379ca3-a0a6-4771-a2b7-469180ba520f\" (UID: \"b2379ca3-a0a6-4771-a2b7-469180ba520f\") "
Sep 10 15:18:10.631960 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:10.631936    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2379ca3-a0a6-4771-a2b7-469180ba520f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2379ca3-a0a6-4771-a2b7-469180ba520f" (UID: "b2379ca3-a0a6-4771-a2b7-469180ba520f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:18:10.732498 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:10.732419    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2379ca3-a0a6-4771-a2b7-469180ba520f-maas-api-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:18:11.416108 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:11.416084    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"
Sep 10 15:18:11.416256 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:11.416083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt" event={"ID":"b2379ca3-a0a6-4771-a2b7-469180ba520f","Type":"ContainerDied","Data":"7414525e34434848378d1c83f14faea5716d67f7cb430392db91db7f91de40bd"}
Sep 10 15:18:11.416256 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:11.416204    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7414525e34434848378d1c83f14faea5716d67f7cb430392db91db7f91de40bd"
Sep 10 15:18:18.460706 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:18.460670    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"]
Sep 10 15:18:18.471657 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:18.471622    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d5a2df-h9htt"]
Sep 10 15:18:19.976451 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:19.976410    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2379ca3-a0a6-4771-a2b7-469180ba520f" path="/var/lib/kubelet/pods/b2379ca3-a0a6-4771-a2b7-469180ba520f/volumes"
Sep 10 15:18:20.960702 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:20.960666    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:18:29.165016 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:18:29.164980    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:19:50.752914 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:19:50.752880    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:20:14.458458 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:20:14.458417    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:20:22.257261 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:20:22.257231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:20:29.355141 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:20:29.355097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:20:36.855293 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:20:36.855255    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:20:45.504895 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:20:45.504801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:20:52.750705 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:20:52.750666    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:21:00.055510 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:21:00.055466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:21:08.060268 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:21:08.060226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:21:46.240038 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:21:46.240001    2565 scope.go:117] "RemoveContainer" containerID="68e418a4b9be5e752b0977e72b5b99a9d7ec5b4b9dfe441e8c233559402b94c3"
Sep 10 15:22:38.862543 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:22:38.862507    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:22:45.357621 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:22:45.357580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:22:54.851275 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:22:54.851238    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:23:07.437861 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.437821    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"]
Sep 10 15:23:07.438249 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.438197    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2379ca3-a0a6-4771-a2b7-469180ba520f" containerName="revoke-keys"
Sep 10 15:23:07.438249 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.438212    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2379ca3-a0a6-4771-a2b7-469180ba520f" containerName="revoke-keys"
Sep 10 15:23:07.438323 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.438283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2379ca3-a0a6-4771-a2b7-469180ba520f" containerName="revoke-keys"
Sep 10 15:23:07.441260 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.441240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.443561 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.443537    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-15d93b6e-kserve-self-signed-certs\""
Sep 10 15:23:07.450501 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.450475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"]
Sep 10 15:23:07.591904 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.591864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-model-cache\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.591904 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.591904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tls-certs\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.592145 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.591928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qs2p\" (UniqueName: \"kubernetes.io/projected/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-kube-api-access-7qs2p\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.592145 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.591950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-home\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.592145 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.591988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tmp-dir\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.592145 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.592062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-dshm\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.692934 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.692833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-home\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.692934 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.692877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tmp-dir\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.693183 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.692940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-dshm\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.693183 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.693032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-model-cache\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.693183 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.693059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tls-certs\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.693183 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.693092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qs2p\" (UniqueName: \"kubernetes.io/projected/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-kube-api-access-7qs2p\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.693385 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.693295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-home\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.693385 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.693314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tmp-dir\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.693385 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.693373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-model-cache\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.695240 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.695220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-dshm\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.695632 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.695610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tls-certs\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.701974 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.701954    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qs2p\" (UniqueName: \"kubernetes.io/projected/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-kube-api-access-7qs2p\") pod \"e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") " pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:07.752143 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:07.752100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:08.025346 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:08.025316    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"]
Sep 10 15:23:08.027460 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:23:08.027435    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09bf5295_5f73_45a3_8d9d_3715cc3d48ad.slice/crio-88cea8bcb6fb04e13308d46ea22cde9d28fd220722ee7643c36fbb393a880a35 WatchSource:0}: Error finding container 88cea8bcb6fb04e13308d46ea22cde9d28fd220722ee7643c36fbb393a880a35: Status 404 returned error can't find the container with id 88cea8bcb6fb04e13308d46ea22cde9d28fd220722ee7643c36fbb393a880a35
Sep 10 15:23:08.994528 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:08.994488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2" event={"ID":"09bf5295-5f73-45a3-8d9d-3715cc3d48ad","Type":"ContainerStarted","Data":"acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196"}
Sep 10 15:23:08.995008 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:08.994539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2" event={"ID":"09bf5295-5f73-45a3-8d9d-3715cc3d48ad","Type":"ContainerStarted","Data":"88cea8bcb6fb04e13308d46ea22cde9d28fd220722ee7643c36fbb393a880a35"}
Sep 10 15:23:09.017022 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:09.016964    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2" podStartSLOduration=2.01694787 podStartE2EDuration="2.01694787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:23:09.013925919 +0000 UTC m=+2123.583465178" watchObservedRunningTime="2026-09-10 15:23:09.01694787 +0000 UTC m=+2123.586487129"
Sep 10 15:23:10.065373 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:10.065331    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:23:17.753111 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:17.753067    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:17.753523 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:17.753158    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:17.765788 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:17.765763    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:18.051972 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:18.051882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:23:42.658228 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:23:42.658142    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:24:16.060914 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:16.060876    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"]
Sep 10 15:24:19.158339 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:19.158304    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:24:26.765483 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:26.765441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:24:31.080789 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.080729    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2" podUID="09bf5295-5f73-45a3-8d9d-3715cc3d48ad" containerName="main" containerID="cri-o://acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196" gracePeriod=45
Sep 10 15:24:31.324630 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.324607    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:24:31.425358 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.425317    2565 generic.go:356] "Generic (PLEG): container finished" podID="09bf5295-5f73-45a3-8d9d-3715cc3d48ad" containerID="acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196" exitCode=0
Sep 10 15:24:31.425358 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.425362    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"
Sep 10 15:24:31.425565 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.425404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2" event={"ID":"09bf5295-5f73-45a3-8d9d-3715cc3d48ad","Type":"ContainerDied","Data":"acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196"}
Sep 10 15:24:31.425565 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.425448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2" event={"ID":"09bf5295-5f73-45a3-8d9d-3715cc3d48ad","Type":"ContainerDied","Data":"88cea8bcb6fb04e13308d46ea22cde9d28fd220722ee7643c36fbb393a880a35"}
Sep 10 15:24:31.425565 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.425467    2565 scope.go:117] "RemoveContainer" containerID="acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196"
Sep 10 15:24:31.434043 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.434027    2565 scope.go:117] "RemoveContainer" containerID="acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196"
Sep 10 15:24:31.434294 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:24:31.434278    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196\": container with ID starting with acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196 not found: ID does not exist" containerID="acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196"
Sep 10 15:24:31.434349 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.434303    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196"} err="failed to get container status \"acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196\": rpc error: code = NotFound desc = could not find container \"acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196\": container with ID starting with acc2fcab76cde9e4a76be40da88d114f3fa4a6885ee7670c5beedd5b0f6bd196 not found: ID does not exist"
Sep 10 15:24:31.449704 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.449683    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-home\") pod \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") "
Sep 10 15:24:31.449812 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.449743    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-model-cache\") pod \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") "
Sep 10 15:24:31.449812 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.449763    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qs2p\" (UniqueName: \"kubernetes.io/projected/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-kube-api-access-7qs2p\") pod \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") "
Sep 10 15:24:31.449812 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.449785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tls-certs\") pod \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") "
Sep 10 15:24:31.449985 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.449814    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-dshm\") pod \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") "
Sep 10 15:24:31.449985 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.449857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tmp-dir\") pod \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\" (UID: \"09bf5295-5f73-45a3-8d9d-3715cc3d48ad\") "
Sep 10 15:24:31.450089 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.450017    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-home" (OuterVolumeSpecName: "home") pod "09bf5295-5f73-45a3-8d9d-3715cc3d48ad" (UID: "09bf5295-5f73-45a3-8d9d-3715cc3d48ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:24:31.450155 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.450131    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "09bf5295-5f73-45a3-8d9d-3715cc3d48ad" (UID: "09bf5295-5f73-45a3-8d9d-3715cc3d48ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:24:31.450214 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.450183    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-home\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:31.450266 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.450209    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09bf5295-5f73-45a3-8d9d-3715cc3d48ad" (UID: "09bf5295-5f73-45a3-8d9d-3715cc3d48ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:24:31.451808 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.451789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-kube-api-access-7qs2p" (OuterVolumeSpecName: "kube-api-access-7qs2p") pod "09bf5295-5f73-45a3-8d9d-3715cc3d48ad" (UID: "09bf5295-5f73-45a3-8d9d-3715cc3d48ad"). InnerVolumeSpecName "kube-api-access-7qs2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:24:31.451936 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.451919    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-dshm" (OuterVolumeSpecName: "dshm") pod "09bf5295-5f73-45a3-8d9d-3715cc3d48ad" (UID: "09bf5295-5f73-45a3-8d9d-3715cc3d48ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:24:31.452036 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.452013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09bf5295-5f73-45a3-8d9d-3715cc3d48ad" (UID: "09bf5295-5f73-45a3-8d9d-3715cc3d48ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:24:31.551657 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.551606    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tls-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:31.551657 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.551630    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-dshm\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:31.551657 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.551660    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-tmp-dir\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:31.551872 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.551670    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-model-cache\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:31.551872 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.551682    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qs2p\" (UniqueName: \"kubernetes.io/projected/09bf5295-5f73-45a3-8d9d-3715cc3d48ad-kube-api-access-7qs2p\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:31.748632 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.748599    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"]
Sep 10 15:24:31.750990 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.750969    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-15d93b6e-kserve-58f8589755-ggmk2"]
Sep 10 15:24:31.976988 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:31.976956    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09bf5295-5f73-45a3-8d9d-3715cc3d48ad" path="/var/lib/kubelet/pods/09bf5295-5f73-45a3-8d9d-3715cc3d48ad/volumes"
Sep 10 15:24:32.552588 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:32.552547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:24:39.957315 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:39.957276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:24:40.314246 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.314166    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"]
Sep 10 15:24:40.314527 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.314516    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09bf5295-5f73-45a3-8d9d-3715cc3d48ad" containerName="main"
Sep 10 15:24:40.314571 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.314529    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09bf5295-5f73-45a3-8d9d-3715cc3d48ad" containerName="main"
Sep 10 15:24:40.314607 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.314586    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09bf5295-5f73-45a3-8d9d-3715cc3d48ad" containerName="main"
Sep 10 15:24:40.318908 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.318891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.321884 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.321855    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:24:40.322653 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.322625    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:24:40.323320 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.323305    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:24:40.324944 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.324924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 15:24:40.335618 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.335591    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vh2ph\""
Sep 10 15:24:40.338923 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.338899    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"]
Sep 10 15:24:40.432371 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.432334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-metrics-certs\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.432556 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.432391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-maas-api-tls\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.432556 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.432480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10854e-5804-4703-9b19-bff47ed28cd3-openshift-service-ca\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.432556 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.432521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmm6w\" (UniqueName: \"kubernetes.io/projected/9e10854e-5804-4703-9b19-bff47ed28cd3-kube-api-access-pmm6w\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.533085 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.533055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-maas-api-tls\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.533259 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.533107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10854e-5804-4703-9b19-bff47ed28cd3-openshift-service-ca\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.533259 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.533136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmm6w\" (UniqueName: \"kubernetes.io/projected/9e10854e-5804-4703-9b19-bff47ed28cd3-kube-api-access-pmm6w\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.533259 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.533175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-metrics-certs\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.533850 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.533822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10854e-5804-4703-9b19-bff47ed28cd3-openshift-service-ca\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.535609 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.535577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-maas-api-tls\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.535717 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.535629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-metrics-certs\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.541717 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.541695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmm6w\" (UniqueName: \"kubernetes.io/projected/9e10854e-5804-4703-9b19-bff47ed28cd3-kube-api-access-pmm6w\") pod \"maas-api-9d7854d48-85lgn\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") " pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.628620 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.628585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:40.907812 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.907777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"]
Sep 10 15:24:40.909183 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:24:40.909154    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e10854e_5804_4703_9b19_bff47ed28cd3.slice/crio-9ac3475f314533508247bf717b7c0cc67a6dba1df8a43708b2565954821285ff WatchSource:0}: Error finding container 9ac3475f314533508247bf717b7c0cc67a6dba1df8a43708b2565954821285ff: Status 404 returned error can't find the container with id 9ac3475f314533508247bf717b7c0cc67a6dba1df8a43708b2565954821285ff
Sep 10 15:24:40.910476 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:40.910460    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:24:41.482822 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:41.482785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn" event={"ID":"9e10854e-5804-4703-9b19-bff47ed28cd3","Type":"ContainerStarted","Data":"62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a"}
Sep 10 15:24:41.483165 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:41.482839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn" event={"ID":"9e10854e-5804-4703-9b19-bff47ed28cd3","Type":"ContainerStarted","Data":"9ac3475f314533508247bf717b7c0cc67a6dba1df8a43708b2565954821285ff"}
Sep 10 15:24:41.483165 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:41.482863    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:41.502785 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:41.502733    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn" podStartSLOduration=1.014469221 podStartE2EDuration="1.50271669s" totalImagesPullingTime="488.247469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:41.501066367 +0000 UTC m=+2216.070605645" watchObservedRunningTime="2026-09-10 15:24:41.50271669 +0000 UTC m=+2216.072255949"
Sep 10 15:24:46.275172 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:46.275136    2565 scope.go:117] "RemoveContainer" containerID="11e9381b8ac19f3ca2e4c8b95366be543c35b48367b68bcb87c1edcce46457c4"
Sep 10 15:24:47.493411 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:47.493381    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:47.563027 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:47.562996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:24:48.135820 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.135784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"]
Sep 10 15:24:48.140397 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.140380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.150768 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.150743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"]
Sep 10 15:24:48.200516 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.200489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dvr\" (UniqueName: \"kubernetes.io/projected/905216ab-542f-4af3-a113-02802e1eb696-kube-api-access-22dvr\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.200667 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.200545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/905216ab-542f-4af3-a113-02802e1eb696-openshift-service-ca\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.200667 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.200617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/905216ab-542f-4af3-a113-02802e1eb696-maas-api-tls\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.200667 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.200660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/905216ab-542f-4af3-a113-02802e1eb696-metrics-certs\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.301847 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.301822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/905216ab-542f-4af3-a113-02802e1eb696-openshift-service-ca\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.301962 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.301873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/905216ab-542f-4af3-a113-02802e1eb696-maas-api-tls\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.301962 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.301897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/905216ab-542f-4af3-a113-02802e1eb696-metrics-certs\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.301962 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.301923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dvr\" (UniqueName: \"kubernetes.io/projected/905216ab-542f-4af3-a113-02802e1eb696-kube-api-access-22dvr\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.302434 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.302413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/905216ab-542f-4af3-a113-02802e1eb696-openshift-service-ca\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.304326 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.304303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/905216ab-542f-4af3-a113-02802e1eb696-maas-api-tls\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.304432 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.304308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/905216ab-542f-4af3-a113-02802e1eb696-metrics-certs\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.309431 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.309406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dvr\" (UniqueName: \"kubernetes.io/projected/905216ab-542f-4af3-a113-02802e1eb696-kube-api-access-22dvr\") pod \"maas-api-5cc85db8b4-pzw2n\" (UID: \"905216ab-542f-4af3-a113-02802e1eb696\") " pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.451541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.451481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:48.722843 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:48.722818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"]
Sep 10 15:24:48.725159 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:24:48.725130    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905216ab_542f_4af3_a113_02802e1eb696.slice/crio-0db4adb9668d9e756d82d118fe13937759f41105a700355f1928be90ce6f7076 WatchSource:0}: Error finding container 0db4adb9668d9e756d82d118fe13937759f41105a700355f1928be90ce6f7076: Status 404 returned error can't find the container with id 0db4adb9668d9e756d82d118fe13937759f41105a700355f1928be90ce6f7076
Sep 10 15:24:49.532424 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:49.532386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n" event={"ID":"905216ab-542f-4af3-a113-02802e1eb696","Type":"ContainerStarted","Data":"c2782d54663ce5560c0cd9cb569246eab8cd4e2dc35cfb7b6d142c9ac4841979"}
Sep 10 15:24:49.532591 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:49.532448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:49.532591 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:49.532472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n" event={"ID":"905216ab-542f-4af3-a113-02802e1eb696","Type":"ContainerStarted","Data":"0db4adb9668d9e756d82d118fe13937759f41105a700355f1928be90ce6f7076"}
Sep 10 15:24:49.553847 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:49.553799    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n" podStartSLOduration=1.188924253 podStartE2EDuration="1.55378482s" totalImagesPullingTime="364.860567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:49.551098234 +0000 UTC m=+2224.120637492" watchObservedRunningTime="2026-09-10 15:24:49.55378482 +0000 UTC m=+2224.123324077"
Sep 10 15:24:54.459202 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:54.459167    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:24:55.543554 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:55.543520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cc85db8b4-pzw2n"
Sep 10 15:24:55.596868 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:55.596839    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"]
Sep 10 15:24:55.597058 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:55.597022    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn" podUID="9e10854e-5804-4703-9b19-bff47ed28cd3" containerName="maas-api" containerID="cri-o://62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a" gracePeriod=30
Sep 10 15:24:56.833010 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.832984    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:56.882379 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.882353    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10854e-5804-4703-9b19-bff47ed28cd3-openshift-service-ca\") pod \"9e10854e-5804-4703-9b19-bff47ed28cd3\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") "
Sep 10 15:24:56.882531 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.882406    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmm6w\" (UniqueName: \"kubernetes.io/projected/9e10854e-5804-4703-9b19-bff47ed28cd3-kube-api-access-pmm6w\") pod \"9e10854e-5804-4703-9b19-bff47ed28cd3\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") "
Sep 10 15:24:56.882531 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.882429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-metrics-certs\") pod \"9e10854e-5804-4703-9b19-bff47ed28cd3\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") "
Sep 10 15:24:56.882668 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.882548    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-maas-api-tls\") pod \"9e10854e-5804-4703-9b19-bff47ed28cd3\" (UID: \"9e10854e-5804-4703-9b19-bff47ed28cd3\") "
Sep 10 15:24:56.882797 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.882768    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e10854e-5804-4703-9b19-bff47ed28cd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e10854e-5804-4703-9b19-bff47ed28cd3" (UID: "9e10854e-5804-4703-9b19-bff47ed28cd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:24:56.882981 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.882958    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e10854e-5804-4703-9b19-bff47ed28cd3-openshift-service-ca\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:56.884595 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.884562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e10854e-5804-4703-9b19-bff47ed28cd3-kube-api-access-pmm6w" (OuterVolumeSpecName: "kube-api-access-pmm6w") pod "9e10854e-5804-4703-9b19-bff47ed28cd3" (UID: "9e10854e-5804-4703-9b19-bff47ed28cd3"). InnerVolumeSpecName "kube-api-access-pmm6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:24:56.884700 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.884618    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e10854e-5804-4703-9b19-bff47ed28cd3" (UID: "9e10854e-5804-4703-9b19-bff47ed28cd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:24:56.884745 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.884720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e10854e-5804-4703-9b19-bff47ed28cd3" (UID: "9e10854e-5804-4703-9b19-bff47ed28cd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:24:56.983991 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.983927    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmm6w\" (UniqueName: \"kubernetes.io/projected/9e10854e-5804-4703-9b19-bff47ed28cd3-kube-api-access-pmm6w\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:56.983991 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.983953    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-metrics-certs\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:56.983991 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:56.983967    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e10854e-5804-4703-9b19-bff47ed28cd3-maas-api-tls\") on node \"ip-10-0-130-2.ec2.internal\" DevicePath \"\""
Sep 10 15:24:57.577418 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.577384    2565 generic.go:356] "Generic (PLEG): container finished" podID="9e10854e-5804-4703-9b19-bff47ed28cd3" containerID="62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a" exitCode=0
Sep 10 15:24:57.577589 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.577431    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"
Sep 10 15:24:57.577589 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.577456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn" event={"ID":"9e10854e-5804-4703-9b19-bff47ed28cd3","Type":"ContainerDied","Data":"62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a"}
Sep 10 15:24:57.577589 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.577498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9d7854d48-85lgn" event={"ID":"9e10854e-5804-4703-9b19-bff47ed28cd3","Type":"ContainerDied","Data":"9ac3475f314533508247bf717b7c0cc67a6dba1df8a43708b2565954821285ff"}
Sep 10 15:24:57.577589 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.577520    2565 scope.go:117] "RemoveContainer" containerID="62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a"
Sep 10 15:24:57.589955 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.589932    2565 scope.go:117] "RemoveContainer" containerID="62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a"
Sep 10 15:24:57.590313 ip-10-0-130-2 kubenswrapper[2565]: E0910 15:24:57.590284    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a\": container with ID starting with 62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a not found: ID does not exist" containerID="62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a"
Sep 10 15:24:57.590403 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.590329    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a"} err="failed to get container status \"62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a\": rpc error: code = NotFound desc = could not find container \"62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a\": container with ID starting with 62ac00a4447e6d7140f24f86a408bb15c3b2cdb2ce555c94b3750457944afe2a not found: ID does not exist"
Sep 10 15:24:57.605090 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.605066    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"]
Sep 10 15:24:57.608248 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.608229    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-9d7854d48-85lgn"]
Sep 10 15:24:57.976828 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:24:57.976796    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e10854e-5804-4703-9b19-bff47ed28cd3" path="/var/lib/kubelet/pods/9e10854e-5804-4703-9b19-bff47ed28cd3/volumes"
Sep 10 15:25:00.251448 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:00.251404    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:25:07.455283 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:07.455241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2fkpg"]
Sep 10 15:25:15.885040 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:15.884966    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4xmjg_c70f6abc-e0f1-4d53-81eb-02b5981534cb/copy-manifests/0.log"
Sep 10 15:25:15.892969 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:15.892940    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4xmjg_c70f6abc-e0f1-4d53-81eb-02b5981534cb/manager/0.log"
Sep 10 15:25:16.449789 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:16.449756    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cc85db8b4-pzw2n_905216ab-542f-4af3-a113-02802e1eb696/maas-api/0.log"
Sep 10 15:25:16.659833 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:16.659804    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817555-h8hsv_9265e256-20cb-43f6-9774-b6d7b22be0bf/cleanup/0.log"
Sep 10 15:25:18.064800 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:18.064772    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2fkpg_0aa28884-ead1-4b7e-9f36-3f15e866af02/limitador/0.log"
Sep 10 15:25:18.857001 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:18.856910    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-mn89w_21e981f8-6f9c-4c01-b0b9-3c366b9deefb/kube-auth-proxy/0.log"
Sep 10 15:25:19.311469 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:19.311370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674fd9cf96-x9dtk_014b8cd8-b074-48cf-8c18-2a325258fa41/router/0.log"
Sep 10 15:25:19.876824 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:19.876783    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-6hch9_ba799cc0-e5c6-4fc9-90d5-dafde7a20a6d/main/0.log"
Sep 10 15:25:20.093187 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:20.093139    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f55kpg_1c6446fd-b2fb-4363-bc34-3a663fa20a6e/main/0.log"
Sep 10 15:25:20.203780 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:20.203693    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hgrps_8bd9fd97-3922-4c43-8234-693c485af606/main/0.log"
Sep 10 15:25:20.320071 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:20.320037    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zfg9l_9e088bb4-4a11-4a55-86be-d790c45cf930/main/0.log"
Sep 10 15:25:28.878932 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:28.878899    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vbtcs_2fbf0051-5f2c-4ef2-af7b-c00c1b52241d/global-pull-secret-syncer/0.log"
Sep 10 15:25:28.973254 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:28.973220    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dww8c_8cd4b2c2-0f8c-45c7-b712-46810b3c5f88/konnectivity-agent/0.log"
Sep 10 15:25:29.066475 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:29.066452    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-2.ec2.internal_bf2a6a1bfd0dc2f4c5a2b0190766ed9a/haproxy/0.log"
Sep 10 15:25:33.761049 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:33.761015    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2fkpg_0aa28884-ead1-4b7e-9f36-3f15e866af02/limitador/0.log"
Sep 10 15:25:35.575997 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:35.575962    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-8tz98_73b9c56f-b47e-434e-b60a-32c8dc4946cc/cluster-monitoring-operator/0.log"
Sep 10 15:25:35.689207 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:35.689171    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-559fb774d7-fkhg6_7685a9a9-0243-46cb-af68-c7f43466138a/metrics-server/0.log"
Sep 10 15:25:35.760778 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:35.760749    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bgbm_885d3e75-5483-4379-a070-d05ec15fb918/node-exporter/0.log"
Sep 10 15:25:35.790263 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:35.790238    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bgbm_885d3e75-5483-4379-a070-d05ec15fb918/kube-rbac-proxy/0.log"
Sep 10 15:25:35.819278 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:35.819256    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bgbm_885d3e75-5483-4379-a070-d05ec15fb918/init-textfile/0.log"
Sep 10 15:25:36.111019 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.110990    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-nn6sc_0ea71623-bc6b-4d11-b0b5-16e4128abcc6/kube-rbac-proxy-main/0.log"
Sep 10 15:25:36.139614 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.139578    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-nn6sc_0ea71623-bc6b-4d11-b0b5-16e4128abcc6/kube-rbac-proxy-self/0.log"
Sep 10 15:25:36.167727 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.167698    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-nn6sc_0ea71623-bc6b-4d11-b0b5-16e4128abcc6/openshift-state-metrics/0.log"
Sep 10 15:25:36.433314 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.433219    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wl9j6_5e6ec021-6bce-4edd-8161-76d0da70f01a/prometheus-operator/0.log"
Sep 10 15:25:36.462329 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.462299    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wl9j6_5e6ec021-6bce-4edd-8161-76d0da70f01a/kube-rbac-proxy/0.log"
Sep 10 15:25:36.550047 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.550019    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9749f8bd7-zk2hg_a88bb5ae-c13b-493d-b8ad-03158c113d40/telemeter-client/0.log"
Sep 10 15:25:36.597620 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.597590    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9749f8bd7-zk2hg_a88bb5ae-c13b-493d-b8ad-03158c113d40/reload/0.log"
Sep 10 15:25:36.634716 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:36.634679    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9749f8bd7-zk2hg_a88bb5ae-c13b-493d-b8ad-03158c113d40/kube-rbac-proxy/0.log"
Sep 10 15:25:38.981917 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.981883    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"]
Sep 10 15:25:38.982347 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.982227    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e10854e-5804-4703-9b19-bff47ed28cd3" containerName="maas-api"
Sep 10 15:25:38.982347 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.982238    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e10854e-5804-4703-9b19-bff47ed28cd3" containerName="maas-api"
Sep 10 15:25:38.982347 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.982301    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e10854e-5804-4703-9b19-bff47ed28cd3" containerName="maas-api"
Sep 10 15:25:38.985471 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.985453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:38.988508 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.988477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h5vch\"/\"kube-root-ca.crt\""
Sep 10 15:25:38.989463 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.989441    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h5vch\"/\"openshift-service-ca.crt\""
Sep 10 15:25:38.989563 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.989485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h5vch\"/\"default-dockercfg-x2924\""
Sep 10 15:25:38.994431 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:38.994409    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"]
Sep 10 15:25:39.053075 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.053041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-podres\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.053075 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.053080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-sys\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.053292 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.053106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-lib-modules\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.053358 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.053301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-proc\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.053427 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.053406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46sv\" (UniqueName: \"kubernetes.io/projected/52076405-603d-4387-b915-16b686311a91-kube-api-access-s46sv\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.154541 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s46sv\" (UniqueName: \"kubernetes.io/projected/52076405-603d-4387-b915-16b686311a91-kube-api-access-s46sv\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.154760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-podres\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.154760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-sys\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.154760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-lib-modules\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.154760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-proc\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.154760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-sys\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.154760 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154741    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-podres\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.155010 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-proc\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.155010 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.154786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52076405-603d-4387-b915-16b686311a91-lib-modules\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.164637 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.164617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46sv\" (UniqueName: \"kubernetes.io/projected/52076405-603d-4387-b915-16b686311a91-kube-api-access-s46sv\") pod \"perf-node-gather-daemonset-2xspg\" (UID: \"52076405-603d-4387-b915-16b686311a91\") " pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.296705 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.296594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.443021 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.442989    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65dcd86c77-l8ftj_e36309a2-f08c-4f4f-9ca3-7dca8daf359a/console/0.log"
Sep 10 15:25:39.479060 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.479025    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-8dxmr_b46acd3a-1ac6-4390-8297-96d7c384c7e4/download-server/0.log"
Sep 10 15:25:39.566876 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.566791    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"]
Sep 10 15:25:39.569623 ip-10-0-130-2 kubenswrapper[2565]: W0910 15:25:39.569593    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod52076405_603d_4387_b915_16b686311a91.slice/crio-350be47f007da8e75f1ba93aa1c70e1efdb353d98dd6af9e18188ff05794ae7f WatchSource:0}: Error finding container 350be47f007da8e75f1ba93aa1c70e1efdb353d98dd6af9e18188ff05794ae7f: Status 404 returned error can't find the container with id 350be47f007da8e75f1ba93aa1c70e1efdb353d98dd6af9e18188ff05794ae7f
Sep 10 15:25:39.797634 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.797599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg" event={"ID":"52076405-603d-4387-b915-16b686311a91","Type":"ContainerStarted","Data":"adb88d58ae03ac6149c87965659732373731ef1cbb6e5be01f389d949e4676d3"}
Sep 10 15:25:39.797820 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.797756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:39.797820 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.797779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg" event={"ID":"52076405-603d-4387-b915-16b686311a91","Type":"ContainerStarted","Data":"350be47f007da8e75f1ba93aa1c70e1efdb353d98dd6af9e18188ff05794ae7f"}
Sep 10 15:25:39.814117 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:39.814066    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg" podStartSLOduration=1.814051114 podStartE2EDuration="1.814051114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:25:39.812891469 +0000 UTC m=+2274.382430727" watchObservedRunningTime="2026-09-10 15:25:39.814051114 +0000 UTC m=+2274.383590373"
Sep 10 15:25:41.354062 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:41.354032    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v46nj_0c065e49-1594-4143-8c64-d2894da542a5/dns/0.log"
Sep 10 15:25:41.377889 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:41.377868    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v46nj_0c065e49-1594-4143-8c64-d2894da542a5/kube-rbac-proxy/0.log"
Sep 10 15:25:41.454172 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:41.454146    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xzqgt_ae64e0a9-6159-40ff-879c-f7ff9ce3097e/dns-node-resolver/0.log"
Sep 10 15:25:42.004687 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:42.004654    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bc9654f77-hdbls_76338a58-7e63-498b-bee4-c5d25aa8d878/registry/0.log"
Sep 10 15:25:42.029986 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:42.029939    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8d9nt_9f8134df-e311-49b1-bbeb-59171cd288eb/node-ca/0.log"
Sep 10 15:25:43.499960 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:43.499931    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-mn89w_21e981f8-6f9c-4c01-b0b9-3c366b9deefb/kube-auth-proxy/0.log"
Sep 10 15:25:43.647424 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:43.647392    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674fd9cf96-x9dtk_014b8cd8-b074-48cf-8c18-2a325258fa41/router/0.log"
Sep 10 15:25:44.548778 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:44.548752    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ttnzd_9a01ff9d-c190-49e0-8196-405a2df80685/serve-healthcheck-canary/0.log"
Sep 10 15:25:45.018921 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:45.018889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-4clbj_9c05fd50-faf9-4283-b529-529d820501a4/insights-operator/0.log"
Sep 10 15:25:45.043804 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:45.043774    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7khtl_0b6ee187-c103-44b3-80d1-3ce30747905c/kube-rbac-proxy/0.log"
Sep 10 15:25:45.065035 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:45.064981    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7khtl_0b6ee187-c103-44b3-80d1-3ce30747905c/exporter/0.log"
Sep 10 15:25:45.087338 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:45.087294    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7khtl_0b6ee187-c103-44b3-80d1-3ce30747905c/extractor/0.log"
Sep 10 15:25:45.813345 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:45.813319    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h5vch/perf-node-gather-daemonset-2xspg"
Sep 10 15:25:48.272959 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:48.272866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4xmjg_c70f6abc-e0f1-4d53-81eb-02b5981534cb/manager/0.log"
Sep 10 15:25:48.303060 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:48.303038    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759746987c-4xmjg_c70f6abc-e0f1-4d53-81eb-02b5981534cb/copy-manifests/0.log"
Sep 10 15:25:49.718816 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:49.718788    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6469457db4-7tphb_5a527c1f-bc0a-460d-9cff-35d6812eb8b8/manager/0.log"
Sep 10 15:25:49.777195 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:49.777159    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-66dtj_0e275381-d069-47ea-9eaa-680f92d588ef/openshift-lws-operator/0.log"
Sep 10 15:25:50.565343 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:50.565314    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-mn89w_21e981f8-6f9c-4c01-b0b9-3c366b9deefb/kube-auth-proxy/0.log"
Sep 10 15:25:50.692485 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:50.692450    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674fd9cf96-x9dtk_014b8cd8-b074-48cf-8c18-2a325258fa41/router/0.log"
Sep 10 15:25:52.620683 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:52.620633    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7497c74497-mn89w_21e981f8-6f9c-4c01-b0b9-3c366b9deefb/kube-auth-proxy/0.log"
Sep 10 15:25:52.744187 ip-10-0-130-2 kubenswrapper[2565]: I0910 15:25:52.744154    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674fd9cf96-x9dtk_014b8cd8-b074-48cf-8c18-2a325258fa41/router/0.log"